-- Logs begin at Thu 2022-09-15 13:28:22 UTC, end at Thu 2022-09-15 14:40:31 UTC. -- Sep 15 13:55:06.607086 dv-cinder-856971-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 15 13:55:06.616970 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Sep 15 13:55:06 2022] *** Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: nodename: dv-cinder-856971-2-iscsi Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: machine: x86_64 Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: clock source: unix Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: pcre jit disabled Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: detected number of CPU cores: 4 Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: current working directory: / Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: detected binary path: /usr/bin/uwsgi-core Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: your processes number limit is 47680 Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: your memory page size is 4096 bytes Sep 15 13:55:06.619015 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: detected max file descriptor number: 2048 Sep 15 13:55:06.619712 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: lock engine: pthread robust mutexes Sep 15 13:55:06.619712 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: thunder lock: enabled Sep 15 13:55:06.619712 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 15 13:55:06.619712 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 15 13:55:06.657989 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: Python main interpreter initialized at 0x55c2ea9a69c0 Sep 15 13:55:06.657989 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: python threads support enabled Sep 15 13:55:06.657989 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: your server socket listen backlog is limited to 100 connections Sep 15 13:55:06.657989 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: your mercy for graceful operations on workers is 90 seconds Sep 15 13:55:06.658346 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: mapped 403077 bytes (393 KB) for 2 cores Sep 15 13:55:06.658429 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: *** Operational MODE: preforking *** Sep 15 13:55:06.658671 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: *** uWSGI is running in multiple interpreter mode *** Sep 15 13:55:06.658671 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: spawned uWSGI master process (pid: 81642) Sep 15 13:55:06.658872 dv-cinder-856971-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 15 13:55:06.659659 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: spawned uWSGI worker 1 (pid: 81654, cores: 1) Sep 15 13:55:06.659766 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: spawned uWSGI worker 2 (pid: 81655, cores: 1) Sep 15 13:55:06.663775 dv-cinder-856971-2-iscsi devstack@keystone.service[81642]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 15 13:55:10.964552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 15 13:55:10.964552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: from cryptography.utils import int_from_bytes Sep 15 13:55:10.965345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 15 13:55:10.965345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: from cryptography.utils import int_from_bytes Sep 15 13:55:10.965345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 15 13:55:10.965345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: from cryptography.utils import int_from_bytes Sep 15 13:55:10.965847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 15 13:55:10.965847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: from cryptography.utils import int_from_bytes Sep 15 13:55:11.014399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 15 13:55:11.014577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 15 13:55:11.014768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] command line args: [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 15 13:55:11.014936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 15 13:55:11.015026 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 15 13:55:11.015135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 15 13:55:11.015228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ================================================================================ {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 15 13:55:11.015335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] command line args: [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 15 13:55:11.015449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] admin_token = **** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.015607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 15 13:55:11.015756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] config_dir = [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.015849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ================================================================================ {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 15 13:55:11.015935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] admin_token = **** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] config_source = [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] debug = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016370 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] config_dir = [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] config_source = [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] debug = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.016912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] list_limit = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] log_config_append = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.017982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] list_limit = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] log_dir = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] log_config_append = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] log_file = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] log_dir = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.018821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] log_file = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.019912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.020980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.021098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.021214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.021310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.021450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.021533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.021698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.021779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.021945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.022772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.022869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.022981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] publish_errors = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.023166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.023337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.023418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.023534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] publish_errors = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.023616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.023776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.023858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] standard_threads = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.024953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] standard_threads = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] use_journal = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] use_json = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] use_stderr = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] use_syslog = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] use_journal = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] use_json = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.025992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.026072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] use_stderr = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.026194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] use_syslog = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.026279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.026424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 15 13:55:11.026509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.026678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.026765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.026915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.026994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.027205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.027414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.027631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.027731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.027913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.027994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.028149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.028227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.028385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.028464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.028633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.028712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.028880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.028960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.029114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.029193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.029352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.029436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.029584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.029664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.029820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.029910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.030065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.030143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.030300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.030386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.030534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.030620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.030788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.030870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.031031 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.031181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.031257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.031334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.031499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.031583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.031776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.031914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.032965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.033984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.034124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.034203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.034365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.034442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.034608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.034694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.034844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.034923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.035081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.035159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.035308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.035398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.035540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.035618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.035793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.035884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.036940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037479 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.037999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.038116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.038227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.038462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.038542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.038613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.038684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.038793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.038904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.039949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.040082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.040169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.040361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.040443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.040612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.040691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.040842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.040937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.041082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.041162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.041313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.041405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.041568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.041660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.041802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.041896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.042999 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.043080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.043245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.043337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.043493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.043574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.043746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.043865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.043981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.044074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.044217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.044335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.044459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.044553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.044694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.044792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.044926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.045968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.046119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.046206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.046353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.046430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.046588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.046665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.046819 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.046905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.047078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.047171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.047315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.047392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.047560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.047646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.047821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.047902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.048985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.049982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.050129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.050207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.050368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.050451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.050598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.050676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.050844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.050936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.051079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.051157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.051587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.051672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.051867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.051956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.052106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.052228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.052337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] auth.external = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.052417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] auth.external = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.052569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.052648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.052860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.052937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.053092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.053171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.053346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] auth.password = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.053430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] auth.password = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.053578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] auth.token = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.053666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] auth.token = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.053816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.053968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.054057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.054207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.054286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.054455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.054533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.054685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.054772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.054930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] credential.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] credential.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.055994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.056149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.056236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.056393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.056472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.056633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.056711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.056868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.056954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.057098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.057176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.057340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.057420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.057582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.057668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.057813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.057891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.058063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.058149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.058339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.058425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.058579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.058657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.058835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.058912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.059071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.059156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.059320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] federation.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.059397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.059548 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] federation.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.059625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.059805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.059889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.060993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.061071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.061241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.061320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.061474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.061551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.061725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.061802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.061960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.062985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.063137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.063215 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.063368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.063448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.063602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.063698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.063854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.063932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.064089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.064169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.064334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.064412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.064565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.064644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.064799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.064879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.065979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.066069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.066223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.066312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.066465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.066553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.066718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.066807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.066952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.067980 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.068118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.068196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.068355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.068435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.068590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.068668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.068822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.069580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.069666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.069740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.069880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.070117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.070195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.070367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.070447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.070601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.070679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.070837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.070914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.071077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.071210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.071340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.071466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.071592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.071739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.071875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.072001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.072128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.072267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.072394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.072518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.072653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.072779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.072911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.073038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.073164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.073288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.073437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.073571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.073697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.073823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.073949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.074088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.074225 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.074352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.074482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.074606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.074732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.074857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.074992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.075118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.075242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.075368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.075494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.075618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.075778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.075907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.076034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.076161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.076293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.076420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.076553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.076678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.076810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.076943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.077068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.077195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.077330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.077464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.077600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.077725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.077850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.077974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.078107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.078234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.078359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.078484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.078607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.078731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.078863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.078988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.079112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.079238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.079374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.079500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.079634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.079822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.079954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.080079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.080205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.080332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.080471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.080623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.080744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.080868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.080995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.081077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.081258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.081388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.081515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.081641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.081768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.081893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.082027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.082153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.082280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.082407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.082533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.082661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.082795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.082930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.083054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.083180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.083306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.083433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.083567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.083708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.083832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.083958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.084084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.084210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.084346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.084473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.085274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.085412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.085544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.085671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] resource.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.085807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.085935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.086079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] resource.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.086201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.086328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.086464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.086593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.086717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.086851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.086978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.087115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.087249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.087375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.087502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.087586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.087723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.087850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.088057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.088185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.088312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] role.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.088444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.088569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] role.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.088696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] role.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.088830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.088966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] role.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.089091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.089217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.089343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.089470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.089604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.089733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.089860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.089985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.090114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.090241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.090375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.090501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.090637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.090764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.090889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.091016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.091148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.091273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.091400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.091524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.091651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.091795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.091930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.092055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.092179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.092302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.092302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.092545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.092545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.092793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.092919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.093043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.093166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.093289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.093421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.093545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.093669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.093792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.093917 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.094057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.094183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.094306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.094429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.094553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.094676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.094807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.094933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.095056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.095179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.095302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.095425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.095556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.095734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.095859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.095989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.096115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.096240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.096377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.096503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.096629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.096753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.096879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.097006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.097141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.097266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] token.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.097392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.097531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.097658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] token.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.097785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.097917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.098044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.098171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.098295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.098422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.098548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.098682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.098807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.098932 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.099059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.099196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.099323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.099458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.099583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.099730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.099854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.099986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.100108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.100283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.100414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.100542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.100542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.100771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.100899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.101044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.101044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.101290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.101290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.101539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.101671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.101794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.101918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.102041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.connection = **** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.102172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.connection = **** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.102303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.102427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.102550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.102673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.102810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.102942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.103065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.103188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.103311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.103433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.103556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.103738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.103864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.103992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.104118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.104246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.104373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.104511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.104648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.104774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.104901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.105028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.105154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.105298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.105426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.105553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.105679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.105805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.105931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.106067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.106211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.106346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.106475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.106603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.106730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.106865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 15 13:55:11.106991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81655) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 15 13:55:11.107119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81654) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 15 13:55:11.132912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.133147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.133329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.133493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.133654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.133827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.141948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.142179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.170615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.170845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.170934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.171091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.224630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.225706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.273486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.273716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.274781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.275021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.430445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.430676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.430803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.430890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.430993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.431112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.431195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.431333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.431415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.431551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.431636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.431828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.454459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.454697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.454787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.454931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.455924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.456407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.467705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.468157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.480882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.481070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.481299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.481399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.481525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.481610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.481746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.481830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.481967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.482050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.482185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81655) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.482266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.482399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.482614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81654) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 15 13:55:11.498650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.498861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 15 13:55:11.499097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.499205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.499308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 15 13:55:11.499405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 15 13:55:11.499646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.499748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.499870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 15 13:55:11.499954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.500144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.500227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.500407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.500487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.500665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.500919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.510042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.510322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.510596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.510789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.510871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.511067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.511337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.511621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.517816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.518491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.520727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.521017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.521304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.521385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.521594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.521677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.521883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.521966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.522713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.522797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.522988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.523249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.523433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.523515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.523733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.523820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.524013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.524095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.524287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.524368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.524584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.524853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.546941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.547900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.548425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.548712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.549363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.549648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.554199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.554482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.554764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.555408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.555705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.555989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.561108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.561399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.562294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.562582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.566984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.567283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.567576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.567895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.568140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.568234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.568438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.568519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.568714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.568796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.569551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.569651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.570029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.570379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.570656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.570931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.585035 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.585317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.586467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.586745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.589156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.590555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.592401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.592702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.593779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.594074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.596903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.598231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.599991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.601723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.603263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.603567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.603879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.604166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.604444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.604723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.604919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.605216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.605498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.605780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.606055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.606346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.620758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.621047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.621319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.621585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.623186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.623479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.623776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.624043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.628327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.629839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.630778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.631340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.632301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.633284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.634830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.635105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.635244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.635443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.635764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.636059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.636464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.637756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.638041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.638339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.638614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.639004 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.647786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.648066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.648348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.648632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.649015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.649317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.649602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.650684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.650971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.651252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.651533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.651938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.652235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.652512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.665345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.668273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.668423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.671213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.671342 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.672701 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.674141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.674276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.674447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.674843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.675553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.677287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.677568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.677961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.681983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.683448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.685094 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.686353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.686543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.686761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.687037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.687321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.689418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.689814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.690101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.690378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.695095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.695388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.695671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.699052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.699343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.699622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.702491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81655) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.702779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.703064 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.703354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.703632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.703932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.704209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.704497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.704780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.705069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.705352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.705620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.705887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.706154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81655) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.706389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81654) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 15 13:55:11.706678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.706957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.707231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.707501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.707793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.708068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.708343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.708619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.708895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.709186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.709446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.709863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.710252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81654) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 15 13:55:11.740322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55c2ea9a69c0 pid: 81655 (default app) Sep 15 13:55:11.744129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000a7ce-72d0-4501-848c-7b71e42a8c92 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:11.744415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000a7ce-72d0-4501-848c-7b71e42a8c92 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:11.744689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000a7ce-72d0-4501-848c-7b71e42a8c92 None None] PATH_INFO: `/v3` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:11.745226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55c2ea9a69c0 pid: 81654 (default app) Sep 15 13:55:11.748850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1/1] 10.222.0.27 () {50 vars in 791 bytes} [Thu Sep 15 13:55:11 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:18.507139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00cadc4-666a-45e9-975d-d75b68aeff82 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:18.507677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00cadc4-666a-45e9-975d-d75b68aeff82 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:18.507677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00cadc4-666a-45e9-975d-d75b68aeff82 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:18.511923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1/2] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:18 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:18.517068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:18.517584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:18.517678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:18.588844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: INFO dbcounter [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] Registered counter for database keystone Sep 15 13:55:18.607415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG oslo_db.sqlalchemy.engines [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81655) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 15 13:55:18.613498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writer thread running {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 15 13:55:18.792082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG passlib.handlers.bcrypt [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81655) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 15 13:55:18.792469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG passlib.handlers.bcrypt [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81655) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 15 13:55:18.850766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:18.852315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:18.983349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 2/3] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:18 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 467 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 15 13:55:19.411348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00cadc4-666a-45e9-975d-d75b68aeff82 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:19.480153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: INFO dbcounter [None req-d00cadc4-666a-45e9-975d-d75b68aeff82 None None] Registered counter for database keystone Sep 15 13:55:19.501570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG oslo_db.sqlalchemy.engines [None req-d00cadc4-666a-45e9-975d-d75b68aeff82 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81654) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 15 13:55:19.508475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writer thread running {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 15 13:55:19.626089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00cadc4-666a-45e9-975d-d75b68aeff82 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:19.630259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3231285d-cccc-42bd-beda-5abb06437c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:19.631009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231285d-cccc-42bd-beda-5abb06437c06 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:19.631443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231285d-cccc-42bd-beda-5abb06437c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:19.631747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231285d-cccc-42bd-beda-5abb06437c06 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:19.639855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3231285d-cccc-42bd-beda-5abb06437c06 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:19.691699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING keystone.common.rbac_enforcer.enforcer [None req-3231285d-cccc-42bd-beda-5abb06437c06 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 15 13:55:19.692047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3231285d-cccc-42bd-beda-5abb06437c06 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:19.701753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 2/4] 10.222.0.27 () {58 vars in 1324 bytes} [Thu Sep 15 13:55:19 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 15 13:55:19.707665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:19.725469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247fec8f-3f99-46bd-94f0-30fa77af4709 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:19.730032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fbfa9c-73b3-4a59-90e4-be38d7b62ef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:19.731137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbfa9c-73b3-4a59-90e4-be38d7b62ef0 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:19.731422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbfa9c-73b3-4a59-90e4-be38d7b62ef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:19.731721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbfa9c-73b3-4a59-90e4-be38d7b62ef0 admin admin] PATH_INFO: `/v3/projects/f973480c94d94371b4cef62cd36d8410` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:19.733319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fbfa9c-73b3-4a59-90e4-be38d7b62ef0 admin admin] RBAC: Authorizing `identity:get_project(project_id=f973480c94d94371b4cef62cd36d8410)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:19.785983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING keystone.common.rbac_enforcer.enforcer [None req-81fbfa9c-73b3-4a59-90e4-be38d7b62ef0 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 15 13:55:19.786327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fbfa9c-73b3-4a59-90e4-be38d7b62ef0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:19.791739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 3/5] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Sep 15 13:55:19 2022] GET /identity/v3/projects/f973480c94d94371b4cef62cd36d8410 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:21.905725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca07099-2f0b-4fe1-a942-3286a3fc1815 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:21.906290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca07099-2f0b-4fe1-a942-3286a3fc1815 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:21.906290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca07099-2f0b-4fe1-a942-3286a3fc1815 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:21.910386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 3/6] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:21.916477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63979fd6-d4f8-457a-9f5d-77e370399c4d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:21.916915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63979fd6-d4f8-457a-9f5d-77e370399c4d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:21.917032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63979fd6-d4f8-457a-9f5d-77e370399c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:21.936958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-63979fd6-d4f8-457a-9f5d-77e370399c4d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:21.938438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-63979fd6-d4f8-457a-9f5d-77e370399c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:21.946196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 4/7] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:21 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 15 13:55:22.216847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca07099-2f0b-4fe1-a942-3286a3fc1815 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:22.229642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca07099-2f0b-4fe1-a942-3286a3fc1815 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:22.233843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf15ba3-d210-4665-a452-4298be674a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:22.234437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf15ba3-d210-4665-a452-4298be674a8b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:22.234718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf15ba3-d210-4665-a452-4298be674a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:22.234995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf15ba3-d210-4665-a452-4298be674a8b admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:22.243148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf15ba3-d210-4665-a452-4298be674a8b admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:22.244954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf15ba3-d210-4665-a452-4298be674a8b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:22.253543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 4/8] 10.222.0.27 () {58 vars in 1324 bytes} [Thu Sep 15 13:55:22 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 13:55:22.259067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63979fd6-d4f8-457a-9f5d-77e370399c4d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:22.268052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63979fd6-d4f8-457a-9f5d-77e370399c4d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:22.272279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ef42dc-ff16-4b6b-8779-65525a8a2efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:22.272897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef42dc-ff16-4b6b-8779-65525a8a2efb admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:22.273177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef42dc-ff16-4b6b-8779-65525a8a2efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:22.273449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef42dc-ff16-4b6b-8779-65525a8a2efb admin admin] PATH_INFO: `/v3/users/22bd926197084ae38005e05ce6f936ab` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:22.274729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ef42dc-ff16-4b6b-8779-65525a8a2efb admin admin] RBAC: Authorizing `identity:get_user(user_id=22bd926197084ae38005e05ce6f936ab)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:22.276578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ef42dc-ff16-4b6b-8779-65525a8a2efb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:22.278076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 5/9] 10.222.0.27 () {56 vars in 1198 bytes} [Thu Sep 15 13:55:22 2022] GET /identity/v3/users/22bd926197084ae38005e05ce6f936ab => generated 247 bytes in 19 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:24.485185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa00b1-66f7-44db-91d8-ea0558d2651f None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:24.485707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa00b1-66f7-44db-91d8-ea0558d2651f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:24.485707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa00b1-66f7-44db-91d8-ea0558d2651f None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:24.487045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 5/10] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:55:24.493003 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba32653-ce48-472c-aecd-766df1fcf27f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:24.493433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba32653-ce48-472c-aecd-766df1fcf27f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:24.493567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba32653-ce48-472c-aecd-766df1fcf27f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:24.512241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8ba32653-ce48-472c-aecd-766df1fcf27f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:24.513418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8ba32653-ce48-472c-aecd-766df1fcf27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:24.539406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b66bd4a-de41-4feb-a3d7-436c4d0302ea None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:24.539706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b66bd4a-de41-4feb-a3d7-436c4d0302ea None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:24.540007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b66bd4a-de41-4feb-a3d7-436c4d0302ea None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:24.541327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 6/11] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:55:24.546403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:24.546696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:24.546972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:24.590234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG passlib.handlers.bcrypt [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81654) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 15 13:55:24.590533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG passlib.handlers.bcrypt [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81654) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 15 13:55:24.596179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 6/12] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:24.639261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:24.640895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:24.740508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 7/13] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:24.869393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba32653-ce48-472c-aecd-766df1fcf27f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:24.893693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:24.975397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba32653-ce48-472c-aecd-766df1fcf27f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:25.009430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:25.009614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0874ff-4caa-45c4-94c4-7f6e536cb86a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:25.010100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0874ff-4caa-45c4-94c4-7f6e536cb86a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:25.010377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0874ff-4caa-45c4-94c4-7f6e536cb86a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:25.010652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0874ff-4caa-45c4-94c4-7f6e536cb86a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:25.024588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0874ff-4caa-45c4-94c4-7f6e536cb86a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-7b0874ff-4caa-45c4-94c4-7f6e536cb86a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:25.033461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:25.034604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:25.035732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 7/14] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:55:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:25.040543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0874ff-4caa-45c4-94c4-7f6e536cb86a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:25.114034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b922cc2-347b-4e01-b20f-75c149309d9a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:25.117068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:25.122795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:25.142155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0874ff-4caa-45c4-94c4-7f6e536cb86a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:25.152461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80851357-0e6d-4eae-813d-51fc5b0517f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:25.153318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80851357-0e6d-4eae-813d-51fc5b0517f5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:25.153597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80851357-0e6d-4eae-813d-51fc5b0517f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:25.153870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80851357-0e6d-4eae-813d-51fc5b0517f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:25.176846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:25.177466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:25.178414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:25.178709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:25.179621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:25.179767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:25.181143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:25.193147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 8/15] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:25.198835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:25.222702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80851357-0e6d-4eae-813d-51fc5b0517f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:25.224258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80851357-0e6d-4eae-813d-51fc5b0517f5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:25.299326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eccb48-55df-47e9-817f-d1be7de0a25c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:25.325775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 8/16] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:55:25.331365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80851357-0e6d-4eae-813d-51fc5b0517f5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:25.334228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:25.334862 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d77872b-abcb-454b-97d3-53c0fffe106c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:25.335594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d77872b-abcb-454b-97d3-53c0fffe106c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:25.336032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d77872b-abcb-454b-97d3-53c0fffe106c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:25.336346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d77872b-abcb-454b-97d3-53c0fffe106c None admin] PATH_INFO: `/v3/users/22bd926197084ae38005e05ce6f936ab` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:25.337614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d77872b-abcb-454b-97d3-53c0fffe106c None admin] RBAC: Authorizing `identity:get_user(user_id=22bd926197084ae38005e05ce6f936ab)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:25.339065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d77872b-abcb-454b-97d3-53c0fffe106c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:25.340686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 9/17] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:25 2022] GET /identity/v3/users/22bd926197084ae38005e05ce6f936ab => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:25.345873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d77872b-abcb-454b-97d3-53c0fffe106c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:25.360510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:25.429981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80851357-0e6d-4eae-813d-51fc5b0517f5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:25.445657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d77872b-abcb-454b-97d3-53c0fffe106c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:25.462674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:25.463522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:25.463921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:25.464193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:25.481059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af67df0f-14f1-4693-be98-2f44b3b3812d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:25.481682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af67df0f-14f1-4693-be98-2f44b3b3812d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:25.482050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af67df0f-14f1-4693-be98-2f44b3b3812d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:25.482248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af67df0f-14f1-4693-be98-2f44b3b3812d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:25.483071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:25.483359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af67df0f-14f1-4693-be98-2f44b3b3812d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:25.484751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:25.484866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af67df0f-14f1-4693-be98-2f44b3b3812d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:25.486376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 10/18] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:25.492999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af67df0f-14f1-4693-be98-2f44b3b3812d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:55:25.503272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:25.504742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:55:25.505883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:55:25.505883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:55:25.505883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:25.505883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 9/19] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:55:25.510117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:25.567230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:25.577222 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:25.596617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af67df0f-14f1-4693-be98-2f44b3b3812d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:25.612120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb23cc4-a061-486e-8eda-e7dcf4921810 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:25.630155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5369ca70-7732-45ad-af8c-4226cd46f5e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:25.631319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369ca70-7732-45ad-af8c-4226cd46f5e5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:25.631597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369ca70-7732-45ad-af8c-4226cd46f5e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:25.631896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369ca70-7732-45ad-af8c-4226cd46f5e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:25.632621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5369ca70-7732-45ad-af8c-4226cd46f5e5 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.domain.id=default, user.id=22bd926197084ae38005e05ce6f936ab) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:25.632996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5369ca70-7732-45ad-af8c-4226cd46f5e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:25.634441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5369ca70-7732-45ad-af8c-4226cd46f5e5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:25.640959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 11/20] 10.222.0.27 () {56 vars in 1322 bytes} [Thu Sep 15 13:55:25 2022] GET /identity/v3/role_assignments?user.id=22bd926197084ae38005e05ce6f936ab&scope.domain.id=default&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 233 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:25.646850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15083bc-1fca-465b-b5b8-7196dd0831a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:25.647482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15083bc-1fca-465b-b5b8-7196dd0831a4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:25.647801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15083bc-1fca-465b-b5b8-7196dd0831a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:25.648112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15083bc-1fca-465b-b5b8-7196dd0831a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:25.648787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15083bc-1fca-465b-b5b8-7196dd0831a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:25.649094 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15083bc-1fca-465b-b5b8-7196dd0831a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:25.650528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15083bc-1fca-465b-b5b8-7196dd0831a4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:25.670485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 10/21] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:55:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:28.191416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afec5fc-7115-4fbb-9dc0-ab84bc322b6c None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:28.192445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afec5fc-7115-4fbb-9dc0-ab84bc322b6c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:28.192906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afec5fc-7115-4fbb-9dc0-ab84bc322b6c None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:28.195621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 12/22] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:28.200762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb55a47-5a63-40a6-b4e4-05c1c91635f5 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:28.201196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb55a47-5a63-40a6-b4e4-05c1c91635f5 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:28.201328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb55a47-5a63-40a6-b4e4-05c1c91635f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:28.227990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-dfb55a47-5a63-40a6-b4e4-05c1c91635f5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:28.229420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dfb55a47-5a63-40a6-b4e4-05c1c91635f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:28.337604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 11/23] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:28.572091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded9765e-15c9-4ecf-a86a-60d6099b261e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:28.572777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded9765e-15c9-4ecf-a86a-60d6099b261e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:28.573250 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded9765e-15c9-4ecf-a86a-60d6099b261e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:28.575038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 13/24] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:28.579942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e20489-a98f-417e-8b96-d325d5c60bc2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:28.580230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e20489-a98f-417e-8b96-d325d5c60bc2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:28.580818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e20489-a98f-417e-8b96-d325d5c60bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:28.605306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-93e20489-a98f-417e-8b96-d325d5c60bc2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:28.606769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-93e20489-a98f-417e-8b96-d325d5c60bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:28.658019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d97e522-06f5-4d30-9f18-50d5d6fde1c4 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:28.658644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d97e522-06f5-4d30-9f18-50d5d6fde1c4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:28.659132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d97e522-06f5-4d30-9f18-50d5d6fde1c4 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:28.660929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 14/25] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:28.666656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debc2cc2-220e-41c8-ad28-b5228ae3d951 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:28.667183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debc2cc2-220e-41c8-ad28-b5228ae3d951 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:28.667662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debc2cc2-220e-41c8-ad28-b5228ae3d951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:28.669611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:28.700675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-debc2cc2-220e-41c8-ad28-b5228ae3d951 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:28.702359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-debc2cc2-220e-41c8-ad28-b5228ae3d951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:28.709818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 12/26] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:28.710432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e20489-a98f-417e-8b96-d325d5c60bc2 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:28.818104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 15/27] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:28.887399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e20489-a98f-417e-8b96-d325d5c60bc2 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:28.946682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:28.947570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:28.948005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:28.948443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:28.959985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:28.961656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:55:28.984809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:28.985864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:55:28.986833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:55:28.986833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:55:28.986833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:28.988723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 13/28] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:28.996843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debc2cc2-220e-41c8-ad28-b5228ae3d951 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:29.058418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:29.093057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:29.108259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:29.173362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debc2cc2-220e-41c8-ad28-b5228ae3d951 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:29.226435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4501d1-d83b-4d51-a12e-9abd3a895f55 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:29.256398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8e3173-351c-428c-b4fd-fd765d70ce7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:29.257069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8e3173-351c-428c-b4fd-fd765d70ce7b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.257356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8e3173-351c-428c-b4fd-fd765d70ce7b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.257635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8e3173-351c-428c-b4fd-fd765d70ce7b None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.258289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8e3173-351c-428c-b4fd-fd765d70ce7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:29.258592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8e3173-351c-428c-b4fd-fd765d70ce7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:29.259997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8e3173-351c-428c-b4fd-fd765d70ce7b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:29.274461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14ed658-85e8-404d-855e-fe6d73a7a4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:29.275080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ed658-85e8-404d-855e-fe6d73a7a4be None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.275358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ed658-85e8-404d-855e-fe6d73a7a4be None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.275628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ed658-85e8-404d-855e-fe6d73a7a4be None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.276332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14ed658-85e8-404d-855e-fe6d73a7a4be None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:29.277765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14ed658-85e8-404d-855e-fe6d73a7a4be None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:29.293660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 16/29] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:55:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:29.294863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45e33a-c096-4f61-9610-42b9cbd19005 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.295344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45e33a-c096-4f61-9610-42b9cbd19005 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.295627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45e33a-c096-4f61-9610-42b9cbd19005 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.297108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 17/30] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:29.297848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e45e33a-c096-4f61-9610-42b9cbd19005 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:29.417596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e45e33a-c096-4f61-9610-42b9cbd19005 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:29.445213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 14/31] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 15 13:55:29 2022] POST /identity/v3/roles => generated 219 bytes in 387 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:55:29.447324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e504eb9-6990-451c-95f3-d15b352d535d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.447614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e504eb9-6990-451c-95f3-d15b352d535d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.447928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e504eb9-6990-451c-95f3-d15b352d535d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.460222 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0c3573-d44c-4968-b1fe-06200789ac07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:29.461323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0c3573-d44c-4968-b1fe-06200789ac07 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.461793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0c3573-d44c-4968-b1fe-06200789ac07 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.462280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0c3573-d44c-4968-b1fe-06200789ac07 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.465401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:29.479733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9e504eb9-6990-451c-95f3-d15b352d535d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:29.479733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9e504eb9-6990-451c-95f3-d15b352d535d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:29.491045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0c3573-d44c-4968-b1fe-06200789ac07 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3f0c3573-d44c-4968-b1fe-06200789ac07 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:29.496776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:29.497663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:29.498447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:29.500944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 18/32] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:55:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:29.502080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0c3573-d44c-4968-b1fe-06200789ac07 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:29.576190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 15/33] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:29.577089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e504eb9-6990-451c-95f3-d15b352d535d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:29.600254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:29.692324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0c3573-d44c-4968-b1fe-06200789ac07 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:29.729811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e504eb9-6990-451c-95f3-d15b352d535d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:29.738134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c571e3b0-b9b4-471b-b0d4-20437dcbd08b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:29.738924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c571e3b0-b9b4-471b-b0d4-20437dcbd08b None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.739330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c571e3b0-b9b4-471b-b0d4-20437dcbd08b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.739713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c571e3b0-b9b4-471b-b0d4-20437dcbd08b None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.741268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c571e3b0-b9b4-471b-b0d4-20437dcbd08b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:29.742849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c571e3b0-b9b4-471b-b0d4-20437dcbd08b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:29.777298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422a4f2-b028-4879-a167-e3bf90de99e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:29.777918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422a4f2-b028-4879-a167-e3bf90de99e3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.778251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422a4f2-b028-4879-a167-e3bf90de99e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.778543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422a4f2-b028-4879-a167-e3bf90de99e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.779315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422a4f2-b028-4879-a167-e3bf90de99e3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:29.779629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422a4f2-b028-4879-a167-e3bf90de99e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:29.781250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0422a4f2-b028-4879-a167-e3bf90de99e3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:29.797176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 16/34] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:29.798481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d892ba-914e-48ca-967b-1edaefde86de None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.798771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d892ba-914e-48ca-967b-1edaefde86de None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.799051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d892ba-914e-48ca-967b-1edaefde86de None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.800622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 17/35] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:29.807222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c3247-a415-4a9e-9eb9-4d10ba0a843e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:29.807620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c3247-a415-4a9e-9eb9-4d10ba0a843e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:29.807930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c3247-a415-4a9e-9eb9-4d10ba0a843e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:29.829848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c75c3247-a415-4a9e-9eb9-4d10ba0a843e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:29.831210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c75c3247-a415-4a9e-9eb9-4d10ba0a843e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:29.905642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 19/36] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:55:29 2022] POST /identity/v3/projects => generated 301 bytes in 404 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:55:29.906741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c571e3b0-b9b4-471b-b0d4-20437dcbd08b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:29.929454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 18/37] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:29.931675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75c3247-a415-4a9e-9eb9-4d10ba0a843e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:29.948572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:29.996553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:30.079990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75c3247-a415-4a9e-9eb9-4d10ba0a843e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:30.120013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673861ff-22a3-4fa0-8924-b9d81d557895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:30.120638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673861ff-22a3-4fa0-8924-b9d81d557895 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:30.120926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673861ff-22a3-4fa0-8924-b9d81d557895 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:30.121210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673861ff-22a3-4fa0-8924-b9d81d557895 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:30.121897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673861ff-22a3-4fa0-8924-b9d81d557895 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:30.123611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673861ff-22a3-4fa0-8924-b9d81d557895 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:30.150811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c571e3b0-b9b4-471b-b0d4-20437dcbd08b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:30.166053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 19/38] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 15 13:55:29 2022] POST /identity/v3/roles => generated 225 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:55:30.168109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff76e03e-56b9-4c8d-9700-e5fb689c59d0 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:30.168537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff76e03e-56b9-4c8d-9700-e5fb689c59d0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:30.168971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff76e03e-56b9-4c8d-9700-e5fb689c59d0 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:30.174824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 20/39] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:30.177816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fef102-2e3b-49f7-ab43-68c9ec10da75 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:30.178177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fef102-2e3b-49f7-ab43-68c9ec10da75 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:30.178469 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fef102-2e3b-49f7-ab43-68c9ec10da75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:30.196214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c03ef0-c610-4ce0-a01c-e667ea0aff4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:30.196936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c03ef0-c610-4ce0-a01c-e667ea0aff4e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:30.197223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c03ef0-c610-4ce0-a01c-e667ea0aff4e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:30.197554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c03ef0-c610-4ce0-a01c-e667ea0aff4e None admin] PATH_INFO: `/v3/users/22bd926197084ae38005e05ce6f936ab` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:30.198972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c03ef0-c610-4ce0-a01c-e667ea0aff4e None admin] RBAC: Authorizing `identity:get_user(user_id=22bd926197084ae38005e05ce6f936ab)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:30.200193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-76fef102-2e3b-49f7-ab43-68c9ec10da75 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:30.200800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c03ef0-c610-4ce0-a01c-e667ea0aff4e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:30.201445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-76fef102-2e3b-49f7-ab43-68c9ec10da75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:30.202547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 20/40] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:29 2022] GET /identity/v3/users/22bd926197084ae38005e05ce6f936ab => generated 247 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:30.208242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c03ef0-c610-4ce0-a01c-e667ea0aff4e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:30.283772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:30.289328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 21/41] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:30.290653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fef102-2e3b-49f7-ab43-68c9ec10da75 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:30.336487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c03ef0-c610-4ce0-a01c-e667ea0aff4e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:30.352474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:30.380332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9285fc3-5976-463d-88ae-ff4001a3c1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:30.381717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9285fc3-5976-463d-88ae-ff4001a3c1bc None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:30.381945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9285fc3-5976-463d-88ae-ff4001a3c1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:30.382218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9285fc3-5976-463d-88ae-ff4001a3c1bc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:30.384255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9285fc3-5976-463d-88ae-ff4001a3c1bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:30.385701 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9285fc3-5976-463d-88ae-ff4001a3c1bc None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:30.387476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 21/42] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:30.393675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9285fc3-5976-463d-88ae-ff4001a3c1bc None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:30.425021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fef102-2e3b-49f7-ab43-68c9ec10da75 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:30.471558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a09345-16d4-49e9-9925-77ef4b2c9ffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:30.472283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a09345-16d4-49e9-9925-77ef4b2c9ffd None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:30.472568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a09345-16d4-49e9-9925-77ef4b2c9ffd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:30.472848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a09345-16d4-49e9-9925-77ef4b2c9ffd None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:30.473518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a09345-16d4-49e9-9925-77ef4b2c9ffd None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:30.475843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a09345-16d4-49e9-9925-77ef4b2c9ffd None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:30.490815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 22/43] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 15 13:55:30 2022] POST /identity/v3/roles => generated 223 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:55:30.521710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9285fc3-5976-463d-88ae-ff4001a3c1bc None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:30.560006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54caf189-8bcb-4c03-ba33-5cc8167208d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:30.560605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54caf189-8bcb-4c03-ba33-5cc8167208d4 None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:30.560887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54caf189-8bcb-4c03-ba33-5cc8167208d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:30.561169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54caf189-8bcb-4c03-ba33-5cc8167208d4 None admin] PATH_INFO: `/v3/domains/default/users/22bd926197084ae38005e05ce6f936ab/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:30.563191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54caf189-8bcb-4c03-ba33-5cc8167208d4 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=22bd926197084ae38005e05ce6f936ab, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:30.564780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54caf189-8bcb-4c03-ba33-5cc8167208d4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:30.587719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 22/44] 10.222.0.27 () {58 vars in 1361 bytes} [Thu Sep 15 13:55:30 2022] PUT /identity/v3/domains/default/users/22bd926197084ae38005e05ce6f936ab/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 13:55:30.590543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a09345-16d4-49e9-9925-77ef4b2c9ffd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:30.663240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:30.713075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a09345-16d4-49e9-9925-77ef4b2c9ffd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:30.759649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0058c488-d158-4105-82f5-06463aa6d770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:30.760323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0058c488-d158-4105-82f5-06463aa6d770 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:30.760615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0058c488-d158-4105-82f5-06463aa6d770 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:30.760886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0058c488-d158-4105-82f5-06463aa6d770 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:30.762276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0058c488-d158-4105-82f5-06463aa6d770 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:30.763766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0058c488-d158-4105-82f5-06463aa6d770 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:30.765701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 23/45] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:30.771709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54caf189-8bcb-4c03-ba33-5cc8167208d4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:30.922047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:30.979321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54caf189-8bcb-4c03-ba33-5cc8167208d4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:31.013502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73fbd10-022e-4e9b-afa6-cfa21d882721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:31.014121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73fbd10-022e-4e9b-afa6-cfa21d882721 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:31.014402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73fbd10-022e-4e9b-afa6-cfa21d882721 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:31.014678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73fbd10-022e-4e9b-afa6-cfa21d882721 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:31.015343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73fbd10-022e-4e9b-afa6-cfa21d882721 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:31.016829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73fbd10-022e-4e9b-afa6-cfa21d882721 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:31.043214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 23/46] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:55:30 2022] POST /identity/v3/projects => generated 312 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:55:32.767185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b8fd5-5f08-4a25-a4da-a8f10a40bc34 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:32.767767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b8fd5-5f08-4a25-a4da-a8f10a40bc34 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:32.767767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b8fd5-5f08-4a25-a4da-a8f10a40bc34 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:32.769238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 24/47] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:32.774635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ceada2-7378-4cea-a1d7-640b11ed049e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:32.775013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ceada2-7378-4cea-a1d7-640b11ed049e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:32.775303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ceada2-7378-4cea-a1d7-640b11ed049e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:32.794092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-41ceada2-7378-4cea-a1d7-640b11ed049e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:32.795573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-41ceada2-7378-4cea-a1d7-640b11ed049e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:32.861195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 24/48] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:33.133800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08b8fd5-5f08-4a25-a4da-a8f10a40bc34 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:33.207877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2efb3a4-eb4b-41e2-ae3b-a444e92b2c87 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:33.208162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2efb3a4-eb4b-41e2-ae3b-a444e92b2c87 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:33.208943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2efb3a4-eb4b-41e2-ae3b-a444e92b2c87 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:33.210507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 25/49] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:55:33.216395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1c4e1-59a3-4df7-8565-69529c3c088d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:33.216681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1c4e1-59a3-4df7-8565-69529c3c088d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:33.216955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1c4e1-59a3-4df7-8565-69529c3c088d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:33.236400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c5d1c4e1-59a3-4df7-8565-69529c3c088d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:33.237604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c5d1c4e1-59a3-4df7-8565-69529c3c088d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:33.256124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:33.307975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 26/50] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:33.343243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08b8fd5-5f08-4a25-a4da-a8f10a40bc34 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:33.369990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:33.377920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3e498c-cc68-447a-a5fd-188433fde9c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:33.378554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3e498c-cc68-447a-a5fd-188433fde9c6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:33.378838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3e498c-cc68-447a-a5fd-188433fde9c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:33.379114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3e498c-cc68-447a-a5fd-188433fde9c6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:33.390810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3e498c-cc68-447a-a5fd-188433fde9c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3d3e498c-cc68-447a-a5fd-188433fde9c6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:33.397217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:33.398064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:33.398816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 25/51] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:55:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:33.404163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d1c4e1-59a3-4df7-8565-69529c3c088d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:33.502531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d1c4e1-59a3-4df7-8565-69529c3c088d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:33.535052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:33.535654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:33.535967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:33.536249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:33.537629 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:33.537939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:33.539333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:33.547279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 27/52] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:33.553213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3e498c-cc68-447a-a5fd-188433fde9c6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:33.577067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:33.595658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:33.653825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3e498c-cc68-447a-a5fd-188433fde9c6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:33.679396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd4f1b-0bf9-4108-81c8-4a4b65ccfb52 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:33.687104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12452b74-efd0-419c-b7fc-940369a43a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:33.687764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12452b74-efd0-419c-b7fc-940369a43a22 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:33.688047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12452b74-efd0-419c-b7fc-940369a43a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:33.688320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12452b74-efd0-419c-b7fc-940369a43a22 None admin] PATH_INFO: `/v3/users/22bd926197084ae38005e05ce6f936ab` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:33.689689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12452b74-efd0-419c-b7fc-940369a43a22 None admin] RBAC: Authorizing `identity:get_user(user_id=22bd926197084ae38005e05ce6f936ab)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:33.691154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12452b74-efd0-419c-b7fc-940369a43a22 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:33.693722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 26/53] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:33 2022] GET /identity/v3/users/22bd926197084ae38005e05ce6f936ab => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:33.698949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12452b74-efd0-419c-b7fc-940369a43a22 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:33.715529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdde6c5-569a-4111-b5e9-16d0c3e59f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:33.716303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdde6c5-569a-4111-b5e9-16d0c3e59f33 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:33.716585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdde6c5-569a-4111-b5e9-16d0c3e59f33 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:33.716857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdde6c5-569a-4111-b5e9-16d0c3e59f33 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:33.717974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdde6c5-569a-4111-b5e9-16d0c3e59f33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:33.719428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdde6c5-569a-4111-b5e9-16d0c3e59f33 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:33.721022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 28/54] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:33.729124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdde6c5-569a-4111-b5e9-16d0c3e59f33 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:33.763567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:33.807719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12452b74-efd0-419c-b7fc-940369a43a22 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:33.827246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:33.845297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdde6c5-569a-4111-b5e9-16d0c3e59f33 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:33.846678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf5007-8f68-4bd6-89a7-26db9eb8fa72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:33.847382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf5007-8f68-4bd6-89a7-26db9eb8fa72 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:33.847666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf5007-8f68-4bd6-89a7-26db9eb8fa72 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:33.847973 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf5007-8f68-4bd6-89a7-26db9eb8fa72 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:33.849196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edf5007-8f68-4bd6-89a7-26db9eb8fa72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:33.850690 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edf5007-8f68-4bd6-89a7-26db9eb8fa72 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:33.852298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 27/55] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:33.858461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf5007-8f68-4bd6-89a7-26db9eb8fa72 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:33.884749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca3f987-af75-4184-93d3-1c692309081a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:33.885372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3f987-af75-4184-93d3-1c692309081a None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:33.885654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3f987-af75-4184-93d3-1c692309081a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:33.885979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3f987-af75-4184-93d3-1c692309081a None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:33.886582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca3f987-af75-4184-93d3-1c692309081a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:33.888177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca3f987-af75-4184-93d3-1c692309081a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:33.917297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 29/56] 10.222.0.27 () {60 vars in 1140 bytes} [Thu Sep 15 13:55:33 2022] POST /identity/v3/projects => generated 298 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:55:33.971031 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf5007-8f68-4bd6-89a7-26db9eb8fa72 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:34.006632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:34.021252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368915c-5004-4bb7-bac1-35c759bd836f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:34.022263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368915c-5004-4bb7-bac1-35c759bd836f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:34.022773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368915c-5004-4bb7-bac1-35c759bd836f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:34.023774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368915c-5004-4bb7-bac1-35c759bd836f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:34.025211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f368915c-5004-4bb7-bac1-35c759bd836f None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.domain.id=default, user.id=22bd926197084ae38005e05ce6f936ab) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:34.025529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f368915c-5004-4bb7-bac1-35c759bd836f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:34.027782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f368915c-5004-4bb7-bac1-35c759bd836f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:34.041140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 28/57] 10.222.0.27 () {56 vars in 1322 bytes} [Thu Sep 15 13:55:33 2022] GET /identity/v3/role_assignments?user.id=22bd926197084ae38005e05ce6f936ab&scope.domain.id=default&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 530 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:36.170064 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aed05f-67cf-45f7-8db9-42cdd9757f1b None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:36.170575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aed05f-67cf-45f7-8db9-42cdd9757f1b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:36.170678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aed05f-67cf-45f7-8db9-42cdd9757f1b None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:36.171993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 30/58] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:36.177610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bade406-e997-4bde-b6c8-2f99c30d7b4f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:36.178029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bade406-e997-4bde-b6c8-2f99c30d7b4f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:36.178200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bade406-e997-4bde-b6c8-2f99c30d7b4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:36.196035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4bade406-e997-4bde-b6c8-2f99c30d7b4f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:36.197239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4bade406-e997-4bde-b6c8-2f99c30d7b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:36.261995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 29/59] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:36.533928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aed05f-67cf-45f7-8db9-42cdd9757f1b None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:36.632742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aed05f-67cf-45f7-8db9-42cdd9757f1b None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:36.665807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fed9c8-90b4-4aa5-ae2d-efb5b713d250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:36.666431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fed9c8-90b4-4aa5-ae2d-efb5b713d250 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:36.666718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fed9c8-90b4-4aa5-ae2d-efb5b713d250 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:36.666998 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fed9c8-90b4-4aa5-ae2d-efb5b713d250 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:36.668144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fed9c8-90b4-4aa5-ae2d-efb5b713d250 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:36.669607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fed9c8-90b4-4aa5-ae2d-efb5b713d250 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:36.671161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 31/60] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:36.676874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bade406-e997-4bde-b6c8-2f99c30d7b4f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:36.772650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bade406-e997-4bde-b6c8-2f99c30d7b4f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:36.789731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:36.806080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87817367-55d4-492a-860a-ead1ded1f50b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:36.806674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87817367-55d4-492a-860a-ead1ded1f50b None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:36.806952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87817367-55d4-492a-860a-ead1ded1f50b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:36.807222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87817367-55d4-492a-860a-ead1ded1f50b None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:36.807897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87817367-55d4-492a-860a-ead1ded1f50b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:36.809553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87817367-55d4-492a-860a-ead1ded1f50b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:36.834906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 30/61] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:55:36 2022] POST /identity/v3/users => generated 275 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:55:40.139709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25355967-020b-4798-9d76-e04e3b4a222f None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.140217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25355967-020b-4798-9d76-e04e3b4a222f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.140290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25355967-020b-4798-9d76-e04e3b4a222f None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.141781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 32/62] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:40.149163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34d0577-0bb4-4d61-ac4b-519c18e90a6f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.150004 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34d0577-0bb4-4d61-ac4b-519c18e90a6f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.150500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34d0577-0bb4-4d61-ac4b-519c18e90a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.177557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f34d0577-0bb4-4d61-ac4b-519c18e90a6f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:40.179093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f34d0577-0bb4-4d61-ac4b-519c18e90a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:40.273605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 31/63] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:40.344749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5fe23-5f03-4676-9ff8-fce323f993c2 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.345378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5fe23-5f03-4676-9ff8-fce323f993c2 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.345853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5fe23-5f03-4676-9ff8-fce323f993c2 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.347668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 33/64] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:40.353465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3379a01-aca8-4055-851c-4824bdfd5bb2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.353755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3379a01-aca8-4055-851c-4824bdfd5bb2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.354035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3379a01-aca8-4055-851c-4824bdfd5bb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.380644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e3379a01-aca8-4055-851c-4824bdfd5bb2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:40.382028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e3379a01-aca8-4055-851c-4824bdfd5bb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:40.416435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d231a7-a99e-444b-8db1-525c94b2d6bf None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.417021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d231a7-a99e-444b-8db1-525c94b2d6bf None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.417496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d231a7-a99e-444b-8db1-525c94b2d6bf None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.419341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 34/65] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:40.430999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab194e96-0a16-4ac7-943f-45c3c901aac7 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.431559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab194e96-0a16-4ac7-943f-45c3c901aac7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.432081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab194e96-0a16-4ac7-943f-45c3c901aac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.449329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:40.467484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ab194e96-0a16-4ac7-943f-45c3c901aac7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:40.468737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ab194e96-0a16-4ac7-943f-45c3c901aac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:40.485961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 32/66] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:40.543507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 35/67] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:40.646168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3379a01-aca8-4055-851c-4824bdfd5bb2 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:40.707421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:40.796224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3379a01-aca8-4055-851c-4824bdfd5bb2 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:40.816129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef39eb0-77b8-459b-8e3a-eba9d1dd2c80 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.816733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef39eb0-77b8-459b-8e3a-eba9d1dd2c80 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.817231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef39eb0-77b8-459b-8e3a-eba9d1dd2c80 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.818918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 36/68] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:40.824325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd830812-0183-4f2a-885a-1165d9e6017e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.824875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd830812-0183-4f2a-885a-1165d9e6017e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.825362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd830812-0183-4f2a-885a-1165d9e6017e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.841045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac700a6-e0dc-4272-b789-b2ba7f677914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:40.842191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac700a6-e0dc-4272-b789-b2ba7f677914 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.842612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac700a6-e0dc-4272-b789-b2ba7f677914 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.843057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac700a6-e0dc-4272-b789-b2ba7f677914 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.846385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-dd830812-0183-4f2a-885a-1165d9e6017e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:40.847831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dd830812-0183-4f2a-885a-1165d9e6017e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:40.855667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac700a6-e0dc-4272-b789-b2ba7f677914 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-4ac700a6-e0dc-4272-b789-b2ba7f677914 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:40.861715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:40.862559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:40.863336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:40.863972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 33/69] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:40.868438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac700a6-e0dc-4272-b789-b2ba7f677914 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:40.921073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 37/70] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:40.926697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d744a9-6d2b-4d9e-acda-6927bf47d1de None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:40.927983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d744a9-6d2b-4d9e-acda-6927bf47d1de None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:40.928724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d744a9-6d2b-4d9e-acda-6927bf47d1de None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:40.934874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 38/71] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:40 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:40.934874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d744a9-6d2b-4d9e-acda-6927bf47d1de None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:40.971437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:40.978618 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac700a6-e0dc-4272-b789-b2ba7f677914 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.011544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.015105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd74ccb-779a-431c-a1d3-19a332ddc020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.015781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd74ccb-779a-431c-a1d3-19a332ddc020 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.016073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd74ccb-779a-431c-a1d3-19a332ddc020 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.016346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd74ccb-779a-431c-a1d3-19a332ddc020 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.028230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd74ccb-779a-431c-a1d3-19a332ddc020 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-bcd74ccb-779a-431c-a1d3-19a332ddc020 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:41.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:41.033891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:41.034651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 34/72] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:55:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:41.035286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4640a855-6cc5-4904-8a21-0ef4f300d8f8 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.035775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4640a855-6cc5-4904-8a21-0ef4f300d8f8 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.036806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4640a855-6cc5-4904-8a21-0ef4f300d8f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.057159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4640a855-6cc5-4904-8a21-0ef4f300d8f8 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:41.058114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d744a9-6d2b-4d9e-acda-6927bf47d1de None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.058729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4640a855-6cc5-4904-8a21-0ef4f300d8f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:41.095079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.095838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.096180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.096424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.097076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:41.097982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:41.098787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:41.107716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 39/73] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:41.108568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.135394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 35/74] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:41.136293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4640a855-6cc5-4904-8a21-0ef4f300d8f8 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.230060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.241365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce855f5-ea0f-4b43-bdc0-fb1dcc0f8e2b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.248899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4640a855-6cc5-4904-8a21-0ef4f300d8f8 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.272668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.282607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d62bff-feb8-4b39-a546-c27bb8132f03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.283434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d62bff-feb8-4b39-a546-c27bb8132f03 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.283747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d62bff-feb8-4b39-a546-c27bb8132f03 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.284065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d62bff-feb8-4b39-a546-c27bb8132f03 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.285172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.285792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.286125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.286412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.287106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:41.287401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:41.289060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:41.295548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d62bff-feb8-4b39-a546-c27bb8132f03 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:41.297467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 36/75] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:41.298436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-24d62bff-feb8-4b39-a546-c27bb8132f03 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:41.302984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:41.304046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:55:41.304996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:41.310373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 40/76] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:41.310373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d62bff-feb8-4b39-a546-c27bb8132f03 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.411888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d62bff-feb8-4b39-a546-c27bb8132f03 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.419182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ae700a-768f-4ada-b4ba-6aa6548043c7 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.447047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceec935-c4f2-451c-94ff-85499f134cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.447679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceec935-c4f2-451c-94ff-85499f134cfb None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.447989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceec935-c4f2-451c-94ff-85499f134cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.448264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ceec935-c4f2-451c-94ff-85499f134cfb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.449378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ceec935-c4f2-451c-94ff-85499f134cfb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:41.450838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ceec935-c4f2-451c-94ff-85499f134cfb None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:41.452441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 41/77] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:41.453208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceec935-c4f2-451c-94ff-85499f134cfb None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.453375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534732ce-e772-4fbc-b0fe-a543ec08b472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.454006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534732ce-e772-4fbc-b0fe-a543ec08b472 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.454290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534732ce-e772-4fbc-b0fe-a543ec08b472 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.454570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534732ce-e772-4fbc-b0fe-a543ec08b472 None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.472170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534732ce-e772-4fbc-b0fe-a543ec08b472 None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:41.473685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534732ce-e772-4fbc-b0fe-a543ec08b472 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:41.475334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 37/78] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:41.476135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534732ce-e772-4fbc-b0fe-a543ec08b472 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.479734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.511740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.556351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ceec935-c4f2-451c-94ff-85499f134cfb None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.575045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534732ce-e772-4fbc-b0fe-a543ec08b472 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.589821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58b128d-3cab-477c-bf5f-cb6056d6562e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.590460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b128d-3cab-477c-bf5f-cb6056d6562e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.590746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b128d-3cab-477c-bf5f-cb6056d6562e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.591024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b128d-3cab-477c-bf5f-cb6056d6562e None admin] PATH_INFO: `/v3/users/22bd926197084ae38005e05ce6f936ab` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.592198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58b128d-3cab-477c-bf5f-cb6056d6562e None admin] RBAC: Authorizing `identity:get_user(user_id=22bd926197084ae38005e05ce6f936ab)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:41.593649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58b128d-3cab-477c-bf5f-cb6056d6562e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:41.595226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 42/79] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/users/22bd926197084ae38005e05ce6f936ab => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:41.596096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58b128d-3cab-477c-bf5f-cb6056d6562e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.611676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.612307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.612592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.612873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.613540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:41.613848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:41.615807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:41.623527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 38/80] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:41.624342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.688909 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.698444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58b128d-3cab-477c-bf5f-cb6056d6562e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.722275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.724760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c968f1-1940-4277-affc-0f6d9a9c9117 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.734276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d0d7ab-c3e0-4771-9fd3-8b232778aa9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.734907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d0d7ab-c3e0-4771-9fd3-8b232778aa9e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.735192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d0d7ab-c3e0-4771-9fd3-8b232778aa9e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.735467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d0d7ab-c3e0-4771-9fd3-8b232778aa9e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.745548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d0d7ab-c3e0-4771-9fd3-8b232778aa9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d8d0d7ab-c3e0-4771-9fd3-8b232778aa9e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:41.750738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:41.751628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:41.752430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 43/81] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:41.753049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d0d7ab-c3e0-4771-9fd3-8b232778aa9e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.761931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3161e5-55ab-412a-b232-ac87c5702d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.762476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3161e5-55ab-412a-b232-ac87c5702d7a None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.762762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3161e5-55ab-412a-b232-ac87c5702d7a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.763041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3161e5-55ab-412a-b232-ac87c5702d7a None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.763732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3161e5-55ab-412a-b232-ac87c5702d7a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:41.765182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3161e5-55ab-412a-b232-ac87c5702d7a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:41.786262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 39/82] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:55:41 2022] POST /identity/v3/users => generated 289 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:55:41.787253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3161e5-55ab-412a-b232-ac87c5702d7a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:41.864709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d0d7ab-c3e0-4771-9fd3-8b232778aa9e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.905301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e9473e-1901-4234-ab96-62153b9b0c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.906327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e9473e-1901-4234-ab96-62153b9b0c12 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.906887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e9473e-1901-4234-ab96-62153b9b0c12 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.907174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e9473e-1901-4234-ab96-62153b9b0c12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.908583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3161e5-55ab-412a-b232-ac87c5702d7a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:41.937918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.953581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371fe084-f252-4ba8-936e-d23cca4032e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:41.954210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371fe084-f252-4ba8-936e-d23cca4032e2 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:41.954495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371fe084-f252-4ba8-936e-d23cca4032e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:41.954773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371fe084-f252-4ba8-936e-d23cca4032e2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:41.968609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:41.996286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e9473e-1901-4234-ab96-62153b9b0c12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:41.998006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e9473e-1901-4234-ab96-62153b9b0c12 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.034333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371fe084-f252-4ba8-936e-d23cca4032e2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:42.035818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371fe084-f252-4ba8-936e-d23cca4032e2 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.127813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 44/83] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:42.128692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e9473e-1901-4234-ab96-62153b9b0c12 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.160831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 40/84] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:42.162653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371fe084-f252-4ba8-936e-d23cca4032e2 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.200379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:42.223027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:42.253626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e9473e-1901-4234-ab96-62153b9b0c12 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:42.296457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371fe084-f252-4ba8-936e-d23cca4032e2 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:42.308016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0235671b-7b1c-49e4-8124-94a6752de3de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:42.308674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235671b-7b1c-49e4-8124-94a6752de3de None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:42.308963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235671b-7b1c-49e4-8124-94a6752de3de None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:42.311992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0235671b-7b1c-49e4-8124-94a6752de3de None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:42.313253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0235671b-7b1c-49e4-8124-94a6752de3de None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:42.314734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0235671b-7b1c-49e4-8124-94a6752de3de None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.316397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 45/85] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:42.317172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0235671b-7b1c-49e4-8124-94a6752de3de None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.352433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:42.353669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:42.354430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:42.355315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:42.356499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:42.357026 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:42.359898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.371542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 41/86] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:42.373512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.427555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0235671b-7b1c-49e4-8124-94a6752de3de None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:42.462404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02430e3f-84ef-4eef-841f-8bce24518f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:42.463020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02430e3f-84ef-4eef-841f-8bce24518f01 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:42.463310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02430e3f-84ef-4eef-841f-8bce24518f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:42.463592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02430e3f-84ef-4eef-841f-8bce24518f01 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:42.464903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02430e3f-84ef-4eef-841f-8bce24518f01 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:42.468039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02430e3f-84ef-4eef-841f-8bce24518f01 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.469755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 46/87] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:42.470543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02430e3f-84ef-4eef-841f-8bce24518f01 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.487979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:42.496358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:42.512830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2731b1e-dd55-4016-80d3-66f1808a9858 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:42.562272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508efd78-dd53-4c05-849a-0e4f231de4b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:42.564024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508efd78-dd53-4c05-849a-0e4f231de4b5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:42.564535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508efd78-dd53-4c05-849a-0e4f231de4b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:42.565260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508efd78-dd53-4c05-849a-0e4f231de4b5 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:42.567066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508efd78-dd53-4c05-849a-0e4f231de4b5 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:42.570180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508efd78-dd53-4c05-849a-0e4f231de4b5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.572907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 42/88] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:42.573990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508efd78-dd53-4c05-849a-0e4f231de4b5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.591738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02430e3f-84ef-4eef-841f-8bce24518f01 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:42.626952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df659e48-8abb-4c99-be7d-bd2a01dddd0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:42.627570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df659e48-8abb-4c99-be7d-bd2a01dddd0b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:42.627897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df659e48-8abb-4c99-be7d-bd2a01dddd0b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:42.628175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df659e48-8abb-4c99-be7d-bd2a01dddd0b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:42.702321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df659e48-8abb-4c99-be7d-bd2a01dddd0b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:42.703838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df659e48-8abb-4c99-be7d-bd2a01dddd0b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.718022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508efd78-dd53-4c05-849a-0e4f231de4b5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:42.730799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:42.762889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b7393-01e1-4280-a2e7-9a41cf1c00aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:42.763559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b7393-01e1-4280-a2e7-9a41cf1c00aa None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:42.763886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b7393-01e1-4280-a2e7-9a41cf1c00aa None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:42.764164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b7393-01e1-4280-a2e7-9a41cf1c00aa None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:42.766019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58b7393-01e1-4280-a2e7-9a41cf1c00aa None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:42.767575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58b7393-01e1-4280-a2e7-9a41cf1c00aa None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.769311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 43/89] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:42.772438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b7393-01e1-4280-a2e7-9a41cf1c00aa None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.800641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:42.848219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 47/90] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:42.849122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df659e48-8abb-4c99-be7d-bd2a01dddd0b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.894445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b7393-01e1-4280-a2e7-9a41cf1c00aa None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:42.937045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:42.938129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:42.938411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:42.938717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:42.940150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] RBAC: Adding query filter params (role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e, user.id=054bf47407b34e4a875fbfb8ad1c5a1b) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:42.940473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:42.942035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:42.949624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 44/91] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/role_assignments?user.id=054bf47407b34e4a875fbfb8ad1c5a1b&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:42.950998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:42.965816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df659e48-8abb-4c99-be7d-bd2a01dddd0b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:42.986277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:43.006778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:43.007637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:43.008047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:43.008418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:43.009360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e, user.id=22bd926197084ae38005e05ce6f936ab) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:43.009791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:43.011482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:43.018155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 48/92] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/role_assignments?user.id=22bd926197084ae38005e05ce6f936ab&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:43.019436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:43.064609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:43.087387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e01a9b5-b812-4d70-9e46-656a08acea3d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:43.201037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea586d5-81e8-4dde-b285-8eaebcc532a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:43.201591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea586d5-81e8-4dde-b285-8eaebcc532a1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:43.201872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea586d5-81e8-4dde-b285-8eaebcc532a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:43.202149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea586d5-81e8-4dde-b285-8eaebcc532a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:43.238886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84517534-68f2-4c11-8068-1a5de140d675 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:43.282921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751208a2-aff7-4f52-a496-9299a4b37ea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:43.284396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751208a2-aff7-4f52-a496-9299a4b37ea4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:43.285062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751208a2-aff7-4f52-a496-9299a4b37ea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:43.287352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751208a2-aff7-4f52-a496-9299a4b37ea4 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:43.288966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751208a2-aff7-4f52-a496-9299a4b37ea4 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:43.291365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751208a2-aff7-4f52-a496-9299a4b37ea4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:43.293157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 49/93] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:43 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:43.304138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751208a2-aff7-4f52-a496-9299a4b37ea4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:43.380482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea586d5-81e8-4dde-b285-8eaebcc532a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:43.382101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea586d5-81e8-4dde-b285-8eaebcc532a1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:43.413125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:43.527574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751208a2-aff7-4f52-a496-9299a4b37ea4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:43.536011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:43.571736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 45/94] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 622 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:43.577612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea586d5-81e8-4dde-b285-8eaebcc532a1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:43.600279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:43.600907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:43.601194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:43.601481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:43.602474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] RBAC: Adding query filter params (role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e, user.id=054bf47407b34e4a875fbfb8ad1c5a1b) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:43.603200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:43.604703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:43.615656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 50/95] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:43 2022] GET /identity/v3/role_assignments?user.id=054bf47407b34e4a875fbfb8ad1c5a1b&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 259 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:43.745940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea586d5-81e8-4dde-b285-8eaebcc532a1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:43.797920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8421f82e-99c5-4e77-96c2-78d7189d893b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:43.798491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8421f82e-99c5-4e77-96c2-78d7189d893b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:43.798774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8421f82e-99c5-4e77-96c2-78d7189d893b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:43.799048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8421f82e-99c5-4e77-96c2-78d7189d893b None admin] PATH_INFO: `/v3/projects/65e73a0199b148dea188f22c07251e3c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:43.800276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8421f82e-99c5-4e77-96c2-78d7189d893b None admin] RBAC: Authorizing `identity:get_project(project_id=65e73a0199b148dea188f22c07251e3c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:43.802833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8421f82e-99c5-4e77-96c2-78d7189d893b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:43.804815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 46/96] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:43 2022] GET /identity/v3/projects/65e73a0199b148dea188f22c07251e3c => generated 312 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:43.811293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:43.950807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7e40f8-1728-457a-b1cd-043dc42dbe10 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:43.966212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:44.003540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc00807-e00b-4c56-b365-759cf1fe6ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:44.004185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc00807-e00b-4c56-b365-759cf1fe6ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:44.004470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc00807-e00b-4c56-b365-759cf1fe6ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:44.005304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc00807-e00b-4c56-b365-759cf1fe6ea5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:44.006040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc00807-e00b-4c56-b365-759cf1fe6ea5 None admin] RBAC: Adding query filter params (role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=65e73a0199b148dea188f22c07251e3c, user.id=054bf47407b34e4a875fbfb8ad1c5a1b) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:44.006350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc00807-e00b-4c56-b365-759cf1fe6ea5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:44.007815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc00807-e00b-4c56-b365-759cf1fe6ea5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:44.013622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 51/97] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:43 2022] GET /identity/v3/role_assignments?user.id=054bf47407b34e4a875fbfb8ad1c5a1b&scope.project.id=65e73a0199b148dea188f22c07251e3c&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 259 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:44.463266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e569ee32-ce89-4fc5-89d0-3546eb3241dd None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:44.463814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e569ee32-ce89-4fc5-89d0-3546eb3241dd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:44.463814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e569ee32-ce89-4fc5-89d0-3546eb3241dd None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:44.465175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 47/98] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:44.479941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee0846-3be7-4378-80c6-b57d252dc81d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:44.480401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee0846-3be7-4378-80c6-b57d252dc81d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:44.480494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee0846-3be7-4378-80c6-b57d252dc81d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:44.503875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-47ee0846-3be7-4378-80c6-b57d252dc81d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:44.505398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-47ee0846-3be7-4378-80c6-b57d252dc81d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:44.600688 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 52/99] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:44.872232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e569ee32-ce89-4fc5-89d0-3546eb3241dd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:44.897665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec7d88b-371f-4c2a-ae25-81ed38a00f25 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:44.897977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec7d88b-371f-4c2a-ae25-81ed38a00f25 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:44.898230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec7d88b-371f-4c2a-ae25-81ed38a00f25 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:44.899720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 53/100] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:44.904923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19a6d48-c8bd-4a21-b6ff-5771912e0298 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:44.905212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19a6d48-c8bd-4a21-b6ff-5771912e0298 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:44.905556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19a6d48-c8bd-4a21-b6ff-5771912e0298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:44.925350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d19a6d48-c8bd-4a21-b6ff-5771912e0298 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:44.927375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d19a6d48-c8bd-4a21-b6ff-5771912e0298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:44.947038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:45.035334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e569ee32-ce89-4fc5-89d0-3546eb3241dd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:45.040418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 54/101] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:45.086358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040363d2-752b-44a5-81c4-f87dfaab3529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:45.086996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040363d2-752b-44a5-81c4-f87dfaab3529 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:45.087277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040363d2-752b-44a5-81c4-f87dfaab3529 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:45.087551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040363d2-752b-44a5-81c4-f87dfaab3529 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:45.097926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040363d2-752b-44a5-81c4-f87dfaab3529 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-040363d2-752b-44a5-81c4-f87dfaab3529 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:45.108229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:45.109178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:45.109958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 48/102] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:44 2022] GET /identity/v3/roles/reader => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:45.115913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19a6d48-c8bd-4a21-b6ff-5771912e0298 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:45.168397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:45.257151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19a6d48-c8bd-4a21-b6ff-5771912e0298 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:45.293083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:45.293702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:45.293984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:45.294259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:45.294925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] RBAC: Adding query filter params (name=reader) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:45.295230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:45.296707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:45.311575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 55/103] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:45.317236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040363d2-752b-44a5-81c4-f87dfaab3529 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:45.429118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040363d2-752b-44a5-81c4-f87dfaab3529 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:45.441925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:45.456082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:45.469112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb5479-6abd-4788-ac73-32d0293c3c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:45.469747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb5479-6abd-4788-ac73-32d0293c3c0a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:45.470017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb5479-6abd-4788-ac73-32d0293c3c0a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:45.470292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb5479-6abd-4788-ac73-32d0293c3c0a None admin] PATH_INFO: `/v3/users/26c891a8d07c43a5bf3cae8b3acdc726` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:45.487342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cb5479-6abd-4788-ac73-32d0293c3c0a None admin] RBAC: Authorizing `identity:get_user(user_id=26c891a8d07c43a5bf3cae8b3acdc726)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:45.489351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cb5479-6abd-4788-ac73-32d0293c3c0a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:45.491520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 49/104] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:45 2022] GET /identity/v3/users/26c891a8d07c43a5bf3cae8b3acdc726 => generated 289 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:45.501146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb5479-6abd-4788-ac73-32d0293c3c0a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:45.624840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c34b84-d8b3-4c8e-a847-af3d797707db None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:45.627146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:45.663593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb5479-6abd-4788-ac73-32d0293c3c0a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:45.710169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3536b009-0071-4643-8642-0cfcc5a5c6d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:45.710838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3536b009-0071-4643-8642-0cfcc5a5c6d7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:45.711129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3536b009-0071-4643-8642-0cfcc5a5c6d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:45.711410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3536b009-0071-4643-8642-0cfcc5a5c6d7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:45.715730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3536b009-0071-4643-8642-0cfcc5a5c6d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:45.717231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3536b009-0071-4643-8642-0cfcc5a5c6d7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:45.719127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 56/105] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:45.725567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3536b009-0071-4643-8642-0cfcc5a5c6d7 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:45.732385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d215f64e-ee29-4508-bde6-9fd3b3aa347f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:45.733111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d215f64e-ee29-4508-bde6-9fd3b3aa347f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:45.733308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d215f64e-ee29-4508-bde6-9fd3b3aa347f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:45.733878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d215f64e-ee29-4508-bde6-9fd3b3aa347f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:45.810889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:45.903837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d215f64e-ee29-4508-bde6-9fd3b3aa347f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:45.905386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d215f64e-ee29-4508-bde6-9fd3b3aa347f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:45.925631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3536b009-0071-4643-8642-0cfcc5a5c6d7 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:45.993215 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f49277-ff45-4497-9582-0b5d8c7d304a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:45.993773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f49277-ff45-4497-9582-0b5d8c7d304a None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:45.994053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f49277-ff45-4497-9582-0b5d8c7d304a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:45.994331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f49277-ff45-4497-9582-0b5d8c7d304a None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:45.995008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f49277-ff45-4497-9582-0b5d8c7d304a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:45.996503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f49277-ff45-4497-9582-0b5d8c7d304a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:46.045451 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 57/106] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:55:45 2022] POST /identity/v3/projects => generated 302 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:55:46.110831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 50/107] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 611 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:55:46.117413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f49277-ff45-4497-9582-0b5d8c7d304a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:46.156232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:46.277931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f49277-ff45-4497-9582-0b5d8c7d304a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:46.325092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b81d30-6c13-43cf-a216-5273c66d97c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:46.325744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b81d30-6c13-43cf-a216-5273c66d97c4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:46.326031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b81d30-6c13-43cf-a216-5273c66d97c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:46.326307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b81d30-6c13-43cf-a216-5273c66d97c4 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:46.330675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b81d30-6c13-43cf-a216-5273c66d97c4 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:46.332196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b81d30-6c13-43cf-a216-5273c66d97c4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:46.334380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 58/108] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:46 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:46.340286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d215f64e-ee29-4508-bde6-9fd3b3aa347f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:46.444523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:46.481057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d215f64e-ee29-4508-bde6-9fd3b3aa347f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:46.517028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700a6374-0cdd-458e-ab7e-b4c6dd3accb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:46.517970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700a6374-0cdd-458e-ab7e-b4c6dd3accb3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:46.518450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700a6374-0cdd-458e-ab7e-b4c6dd3accb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:46.518942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700a6374-0cdd-458e-ab7e-b4c6dd3accb3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:46.519888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700a6374-0cdd-458e-ab7e-b4c6dd3accb3 None admin] RBAC: Adding query filter params (role.id=9950301e6a2d4ba594bc506ec017aad0, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e, user.id=26c891a8d07c43a5bf3cae8b3acdc726) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:46.520927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700a6374-0cdd-458e-ab7e-b4c6dd3accb3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:46.522621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700a6374-0cdd-458e-ab7e-b4c6dd3accb3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:46.524733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88556f58-4a4b-475d-86bc-34b4ef39c75f None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:46.525025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88556f58-4a4b-475d-86bc-34b4ef39c75f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:46.525478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88556f58-4a4b-475d-86bc-34b4ef39c75f None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:46.526707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 59/109] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:46.530919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 51/110] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:46 2022] GET /identity/v3/role_assignments?user.id=26c891a8d07c43a5bf3cae8b3acdc726&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=9950301e6a2d4ba594bc506ec017aad0 => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:46.543098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3491d39c-6bac-4ded-9f1d-a1e26d499fc5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:46.543394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3491d39c-6bac-4ded-9f1d-a1e26d499fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:46.543669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3491d39c-6bac-4ded-9f1d-a1e26d499fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:46.585342 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3491d39c-6bac-4ded-9f1d-a1e26d499fc5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:46.586569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3491d39c-6bac-4ded-9f1d-a1e26d499fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:46.672686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee9232c-8ada-4c39-a041-943659169b92 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:46.673310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee9232c-8ada-4c39-a041-943659169b92 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:46.673807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee9232c-8ada-4c39-a041-943659169b92 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:46.675679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 52/111] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:46.683208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 60/112] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 15 13:55:46.684318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5e8008-7df3-4153-8a76-667f31cd8bbb None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:46.684811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5e8008-7df3-4153-8a76-667f31cd8bbb None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:46.685309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5e8008-7df3-4153-8a76-667f31cd8bbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:46.706555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4f5e8008-7df3-4153-8a76-667f31cd8bbb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:46.708745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4f5e8008-7df3-4153-8a76-667f31cd8bbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:46.802131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 53/113] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:46.810834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90f57e3-52f8-4feb-b6f6-159f048954d5 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:46.811152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90f57e3-52f8-4feb-b6f6-159f048954d5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:46.811404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90f57e3-52f8-4feb-b6f6-159f048954d5 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:46.812849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 61/114] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:46.825285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991a5b9-696a-4059-ac67-69d00a36dacd None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:46.825580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991a5b9-696a-4059-ac67-69d00a36dacd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:46.825851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3991a5b9-696a-4059-ac67-69d00a36dacd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:46.852921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-3991a5b9-696a-4059-ac67-69d00a36dacd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:46.854150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3991a5b9-696a-4059-ac67-69d00a36dacd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:46.926686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:46.936607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 54/115] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:46.997439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90f57e3-52f8-4feb-b6f6-159f048954d5 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:47.024339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:47.139195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90f57e3-52f8-4feb-b6f6-159f048954d5 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:47.189766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cc3efe-5c28-49be-8fd0-706c09eda2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:47.190406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cc3efe-5c28-49be-8fd0-706c09eda2ed None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.190689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cc3efe-5c28-49be-8fd0-706c09eda2ed None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.190968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cc3efe-5c28-49be-8fd0-706c09eda2ed None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.205865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cc3efe-5c28-49be-8fd0-706c09eda2ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-18cc3efe-5c28-49be-8fd0-706c09eda2ed None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:47.211236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:47.212111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:47.213276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 62/116] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:55:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:47.220792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3991a5b9-696a-4059-ac67-69d00a36dacd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:47.376250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3991a5b9-696a-4059-ac67-69d00a36dacd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:47.376992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cc3efe-5c28-49be-8fd0-706c09eda2ed None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:47.423777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:47.424789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.425279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.426046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.427614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:47.428138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:47.429801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:47.439207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 55/117] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:47.440586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:47.502590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:47.540085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:47.548824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9cde32-353d-4118-a61e-2c9707d72ffd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:47.578869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cc3efe-5c28-49be-8fd0-706c09eda2ed None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:47.584666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65489c32-490e-484b-bfa2-578ce043f3de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:47.585433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65489c32-490e-484b-bfa2-578ce043f3de None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.585781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65489c32-490e-484b-bfa2-578ce043f3de None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.586167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65489c32-490e-484b-bfa2-578ce043f3de None admin] PATH_INFO: `/v3/roles/member` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.597104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65489c32-490e-484b-bfa2-578ce043f3de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-65489c32-490e-484b-bfa2-578ce043f3de None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:47.603109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:47.604020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:47.604788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 56/118] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:47.605426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65489c32-490e-484b-bfa2-578ce043f3de None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:47.628249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77445c8-4aca-4ead-9f07-7b923efbed35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:47.628891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77445c8-4aca-4ead-9f07-7b923efbed35 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.629262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77445c8-4aca-4ead-9f07-7b923efbed35 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.629620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77445c8-4aca-4ead-9f07-7b923efbed35 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.641446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77445c8-4aca-4ead-9f07-7b923efbed35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d77445c8-4aca-4ead-9f07-7b923efbed35 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:47.646824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:47.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:47.648475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 63/119] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:55:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:55:47.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc75a40-1409-49fd-be53-5d00501b2f41 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.654197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc75a40-1409-49fd-be53-5d00501b2f41 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.654506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc75a40-1409-49fd-be53-5d00501b2f41 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.655952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 64/120] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:47 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:47.656663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc75a40-1409-49fd-be53-5d00501b2f41 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:47.719414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65489c32-490e-484b-bfa2-578ce043f3de None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:47.736771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:47.757413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41de6189-15f5-4242-8e0c-d6b77fe07e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:47.758045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de6189-15f5-4242-8e0c-d6b77fe07e85 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.758327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de6189-15f5-4242-8e0c-d6b77fe07e85 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.758601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de6189-15f5-4242-8e0c-d6b77fe07e85 None admin] PATH_INFO: `/v3/users/22bd926197084ae38005e05ce6f936ab` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.759751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41de6189-15f5-4242-8e0c-d6b77fe07e85 None admin] RBAC: Authorizing `identity:get_user(user_id=22bd926197084ae38005e05ce6f936ab)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:47.761215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41de6189-15f5-4242-8e0c-d6b77fe07e85 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:47.762837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 57/121] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:47 2022] GET /identity/v3/users/22bd926197084ae38005e05ce6f936ab => generated 247 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:47.767429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41de6189-15f5-4242-8e0c-d6b77fe07e85 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:47.799678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc75a40-1409-49fd-be53-5d00501b2f41 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:47.841339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f439c56e-544e-4c25-96bd-f488b4155d4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:47.842020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439c56e-544e-4c25-96bd-f488b4155d4a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.842304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439c56e-544e-4c25-96bd-f488b4155d4a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.842635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439c56e-544e-4c25-96bd-f488b4155d4a None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.843381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439c56e-544e-4c25-96bd-f488b4155d4a None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:47.843712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439c56e-544e-4c25-96bd-f488b4155d4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:47.845311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439c56e-544e-4c25-96bd-f488b4155d4a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:47.852842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:47.857759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 65/122] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:47.860088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004bd7fc-395f-42a3-bc9a-e334d3b8282f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.860811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004bd7fc-395f-42a3-bc9a-e334d3b8282f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.861595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004bd7fc-395f-42a3-bc9a-e334d3b8282f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.882879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-004bd7fc-395f-42a3-bc9a-e334d3b8282f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:47.884185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-004bd7fc-395f-42a3-bc9a-e334d3b8282f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:47.896722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41de6189-15f5-4242-8e0c-d6b77fe07e85 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:47.943539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:47.944402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:47.944849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:47.945267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:47.946603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:47.947326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:47.949160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:47.960662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 66/123] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:47.961223 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 58/124] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:55:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:47.965715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004bd7fc-395f-42a3-bc9a-e334d3b8282f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:47.967810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.033488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:48.085143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004bd7fc-395f-42a3-bc9a-e334d3b8282f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.104174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef4c923-5dda-4bbb-9ce3-80c86ef3cb00 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.117462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:48.124993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da67f1f5-cf6c-45fe-ad72-19d8132da259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.125809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da67f1f5-cf6c-45fe-ad72-19d8132da259 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.126238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da67f1f5-cf6c-45fe-ad72-19d8132da259 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.126680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da67f1f5-cf6c-45fe-ad72-19d8132da259 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.127994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da67f1f5-cf6c-45fe-ad72-19d8132da259 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:48.129885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da67f1f5-cf6c-45fe-ad72-19d8132da259 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:48.131759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 67/125] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:47 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:48.132734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da67f1f5-cf6c-45fe-ad72-19d8132da259 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.146637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b106c-ccb2-4b13-a1e4-9b07790aeea3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.147274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b106c-ccb2-4b13-a1e4-9b07790aeea3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.147554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b106c-ccb2-4b13-a1e4-9b07790aeea3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.148305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b106c-ccb2-4b13-a1e4-9b07790aeea3 None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.149239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52b106c-ccb2-4b13-a1e4-9b07790aeea3 None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:48.150911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52b106c-ccb2-4b13-a1e4-9b07790aeea3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:48.152649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 59/126] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:47 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:48.153527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b106c-ccb2-4b13-a1e4-9b07790aeea3 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.245053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da67f1f5-cf6c-45fe-ad72-19d8132da259 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.279447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b106c-ccb2-4b13-a1e4-9b07790aeea3 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.282229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae460214-f0ae-40ee-8241-bd458c03e52f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.283707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae460214-f0ae-40ee-8241-bd458c03e52f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.284228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae460214-f0ae-40ee-8241-bd458c03e52f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.284700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae460214-f0ae-40ee-8241-bd458c03e52f None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.286123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae460214-f0ae-40ee-8241-bd458c03e52f None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:48.288005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae460214-f0ae-40ee-8241-bd458c03e52f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:48.289827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 68/127] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:48 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:48.290830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae460214-f0ae-40ee-8241-bd458c03e52f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.294873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:48.321134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ee14d4-8930-422a-a692-16eebc1196eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.322010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ee14d4-8930-422a-a692-16eebc1196eb None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.322471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ee14d4-8930-422a-a692-16eebc1196eb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.322880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ee14d4-8930-422a-a692-16eebc1196eb None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/users/22bd926197084ae38005e05ce6f936ab/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.325162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ee14d4-8930-422a-a692-16eebc1196eb None admin] RBAC: Authorizing `identity:create_grant(project_id=3d187412d8794cd7a7ad12d6f240c67e, user_id=22bd926197084ae38005e05ce6f936ab, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:48.326944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ee14d4-8930-422a-a692-16eebc1196eb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:48.344064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 60/128] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:55:48 2022] PUT /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/users/22bd926197084ae38005e05ce6f936ab/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:55:48.345299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ee14d4-8930-422a-a692-16eebc1196eb None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.392997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:48.458746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae460214-f0ae-40ee-8241-bd458c03e52f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.499337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de5ccae-5fd4-4f79-9c28-e010d874406f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.500018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5ccae-5fd4-4f79-9c28-e010d874406f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.500357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5ccae-5fd4-4f79-9c28-e010d874406f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.500643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5ccae-5fd4-4f79-9c28-e010d874406f None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.501805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de5ccae-5fd4-4f79-9c28-e010d874406f None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:48.505183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de5ccae-5fd4-4f79-9c28-e010d874406f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:48.505183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 69/129] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:48 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:48.505974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de5ccae-5fd4-4f79-9c28-e010d874406f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.597290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ee14d4-8930-422a-a692-16eebc1196eb None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.617471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de5ccae-5fd4-4f79-9c28-e010d874406f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.655645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d267a994-9a19-4868-93ff-90b4fcf27291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.657281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d267a994-9a19-4868-93ff-90b4fcf27291 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.657281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d267a994-9a19-4868-93ff-90b4fcf27291 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.657281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d267a994-9a19-4868-93ff-90b4fcf27291 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.659328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:48.672755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d267a994-9a19-4868-93ff-90b4fcf27291 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d267a994-9a19-4868-93ff-90b4fcf27291 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:48.678448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:48.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:48.681729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:48.683722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 70/130] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:48.684533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a198e5-6b51-4a40-870e-a75a57ccf5e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.684719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d267a994-9a19-4868-93ff-90b4fcf27291 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.685983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a198e5-6b51-4a40-870e-a75a57ccf5e9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.686810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a198e5-6b51-4a40-870e-a75a57ccf5e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.687518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a198e5-6b51-4a40-870e-a75a57ccf5e9 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.690916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a198e5-6b51-4a40-870e-a75a57ccf5e9 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:48.696691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a198e5-6b51-4a40-870e-a75a57ccf5e9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:48.698750 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 61/131] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:48 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:48.701508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a198e5-6b51-4a40-870e-a75a57ccf5e9 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.782017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:48.812919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d267a994-9a19-4868-93ff-90b4fcf27291 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.884506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e4de8e-d1c4-47f1-aaff-dd87806a84ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.885501 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e4de8e-d1c4-47f1-aaff-dd87806a84ca None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.885975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e4de8e-d1c4-47f1-aaff-dd87806a84ca None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.886458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e4de8e-d1c4-47f1-aaff-dd87806a84ca None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/users/054bf47407b34e4a875fbfb8ad1c5a1b/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.890828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e4de8e-d1c4-47f1-aaff-dd87806a84ca None admin] RBAC: Authorizing `identity:create_grant(project_id=3d187412d8794cd7a7ad12d6f240c67e, user_id=054bf47407b34e4a875fbfb8ad1c5a1b, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:48.892685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e4de8e-d1c4-47f1-aaff-dd87806a84ca None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:48.903192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a198e5-6b51-4a40-870e-a75a57ccf5e9 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:48.930458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 71/132] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:55:48 2022] PUT /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/users/054bf47407b34e4a875fbfb8ad1c5a1b/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:55:48.931812 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e4de8e-d1c4-47f1-aaff-dd87806a84ca None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:48.993207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:48.993846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:48.994138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:48.994420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:48.995110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:48.995425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:48.996854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:49.030333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 62/133] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:49.039770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:49.053026 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:49.118335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e4de8e-d1c4-47f1-aaff-dd87806a84ca None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:49.224200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083f773b-3f60-420b-941d-b2d55926d90c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:49.232421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29064459-faef-4aa1-a5b8-3682ac85c381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:49.232976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:49.233100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29064459-faef-4aa1-a5b8-3682ac85c381 None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:49.233390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29064459-faef-4aa1-a5b8-3682ac85c381 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:49.233671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29064459-faef-4aa1-a5b8-3682ac85c381 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/users/054bf47407b34e4a875fbfb8ad1c5a1b/roles/a0256b6f3d6c4517be88b324fcbc1145` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:49.240141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29064459-faef-4aa1-a5b8-3682ac85c381 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d187412d8794cd7a7ad12d6f240c67e, user_id=054bf47407b34e4a875fbfb8ad1c5a1b, role_id=a0256b6f3d6c4517be88b324fcbc1145)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:49.241625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29064459-faef-4aa1-a5b8-3682ac85c381 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:49.269430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb242084-2c11-47fe-aa96-8fa08b10a360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:49.270154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb242084-2c11-47fe-aa96-8fa08b10a360 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:49.270492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb242084-2c11-47fe-aa96-8fa08b10a360 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:49.270771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb242084-2c11-47fe-aa96-8fa08b10a360 None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:49.272204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb242084-2c11-47fe-aa96-8fa08b10a360 None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:49.273670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb242084-2c11-47fe-aa96-8fa08b10a360 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:49.275605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 63/134] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:49 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:49.277926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 72/135] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:55:48 2022] PUT /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/users/054bf47407b34e4a875fbfb8ad1c5a1b/roles/a0256b6f3d6c4517be88b324fcbc1145 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:55:49.281442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb242084-2c11-47fe-aa96-8fa08b10a360 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:49.329416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48d0fc-117f-43ab-a686-eabe98761561 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:49.329723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48d0fc-117f-43ab-a686-eabe98761561 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:49.329993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b48d0fc-117f-43ab-a686-eabe98761561 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:49.331476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 73/136] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:49.345394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa74356-00c2-4c4c-afcb-67bed7a8dc16 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:49.345720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa74356-00c2-4c4c-afcb-67bed7a8dc16 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:49.346073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa74356-00c2-4c4c-afcb-67bed7a8dc16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:49.379953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6aa74356-00c2-4c4c-afcb-67bed7a8dc16 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:49.381201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6aa74356-00c2-4c4c-afcb-67bed7a8dc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:49.454267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb242084-2c11-47fe-aa96-8fa08b10a360 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:49.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e018aee-24aa-4abf-a52c-bdbded19c8b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:49.495435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e018aee-24aa-4abf-a52c-bdbded19c8b8 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:49.495751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e018aee-24aa-4abf-a52c-bdbded19c8b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:49.496045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e018aee-24aa-4abf-a52c-bdbded19c8b8 None admin] PATH_INFO: `/v3/projects/65e73a0199b148dea188f22c07251e3c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:49.497325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e018aee-24aa-4abf-a52c-bdbded19c8b8 None admin] RBAC: Authorizing `identity:get_project(project_id=65e73a0199b148dea188f22c07251e3c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:49.498770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e018aee-24aa-4abf-a52c-bdbded19c8b8 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:49.500573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 64/137] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:49 2022] GET /identity/v3/projects/65e73a0199b148dea188f22c07251e3c => generated 312 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:49.501936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb1e47-b1d7-4bf0-bf66-b6a89943058a None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:49.502282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb1e47-b1d7-4bf0-bf66-b6a89943058a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:49.502569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb1e47-b1d7-4bf0-bf66-b6a89943058a None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:49.511019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 65/138] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:49 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:49.511952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cb1e47-b1d7-4bf0-bf66-b6a89943058a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:49.568676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:49.585644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 74/139] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:49.586911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24ada9c-b7d3-4774-b8b4-ab6879f3dbc4 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:49.587205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24ada9c-b7d3-4774-b8b4-ab6879f3dbc4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:49.587504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24ada9c-b7d3-4774-b8b4-ab6879f3dbc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:49.608074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e24ada9c-b7d3-4774-b8b4-ab6879f3dbc4 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:49.609334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e24ada9c-b7d3-4774-b8b4-ab6879f3dbc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:49.636063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cb1e47-b1d7-4bf0-bf66-b6a89943058a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:49.671461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:49.678903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f670e43-062c-4f89-b224-784c43571f9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:49.680176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f670e43-062c-4f89-b224-784c43571f9d None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:49.680760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f670e43-062c-4f89-b224-784c43571f9d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:49.681250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f670e43-062c-4f89-b224-784c43571f9d None admin] PATH_INFO: `/v3/projects/65e73a0199b148dea188f22c07251e3c/users/054bf47407b34e4a875fbfb8ad1c5a1b/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:49.683637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f670e43-062c-4f89-b224-784c43571f9d None admin] RBAC: Authorizing `identity:create_grant(project_id=65e73a0199b148dea188f22c07251e3c, user_id=054bf47407b34e4a875fbfb8ad1c5a1b, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:49.685371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f670e43-062c-4f89-b224-784c43571f9d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:49.706957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 66/140] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:55:49 2022] PUT /identity/v3/projects/65e73a0199b148dea188f22c07251e3c/users/054bf47407b34e4a875fbfb8ad1c5a1b/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:55:49.729756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 75/141] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:49.966313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f670e43-062c-4f89-b224-784c43571f9d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:50.088459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f670e43-062c-4f89-b224-784c43571f9d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:50.090364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:50.132864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6720bc1f-53df-4fab-86fb-d7308682ec90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:50.133547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720bc1f-53df-4fab-86fb-d7308682ec90 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:50.133836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720bc1f-53df-4fab-86fb-d7308682ec90 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:50.134118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720bc1f-53df-4fab-86fb-d7308682ec90 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:50.147232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6720bc1f-53df-4fab-86fb-d7308682ec90 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-6720bc1f-53df-4fab-86fb-d7308682ec90 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:50.152082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:50.152908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:50.153832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 67/142] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:50.161631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24ada9c-b7d3-4774-b8b4-ab6879f3dbc4 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:50.231231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6720bc1f-53df-4fab-86fb-d7308682ec90 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:50.340820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24ada9c-b7d3-4774-b8b4-ab6879f3dbc4 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:50.411859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6720bc1f-53df-4fab-86fb-d7308682ec90 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:50.415811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:50.416448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:50.416729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:50.417003 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:50.417689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:50.417994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:50.419412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:50.444779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 76/143] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:50.448713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:50.503185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586896f7-cc58-424d-9dfa-ae044e0aee63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:50.504205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586896f7-cc58-424d-9dfa-ae044e0aee63 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:50.505329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586896f7-cc58-424d-9dfa-ae044e0aee63 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:50.505975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586896f7-cc58-424d-9dfa-ae044e0aee63 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:50.513102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586896f7-cc58-424d-9dfa-ae044e0aee63 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:50.514797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586896f7-cc58-424d-9dfa-ae044e0aee63 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:50.516730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 68/144] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:50.528056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586896f7-cc58-424d-9dfa-ae044e0aee63 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:50.588070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:50.625911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:50.791587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586896f7-cc58-424d-9dfa-ae044e0aee63 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:50.826732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48e909f-e15a-4149-b0cf-d857e4d08da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:50.827347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48e909f-e15a-4149-b0cf-d857e4d08da5 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:50.827633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48e909f-e15a-4149-b0cf-d857e4d08da5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:50.827949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48e909f-e15a-4149-b0cf-d857e4d08da5 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:50.828611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48e909f-e15a-4149-b0cf-d857e4d08da5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:50.830068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48e909f-e15a-4149-b0cf-d857e4d08da5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:50.836546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ac8b39-a02d-4b4b-b333-7e4103e0a635 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:50.860993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 69/145] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:55:50 2022] POST /identity/v3/users => generated 283 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:55:50.916676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5837b0e5-9eae-4ae6-8367-37c451dd6bae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:50.917350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5837b0e5-9eae-4ae6-8367-37c451dd6bae None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:50.917761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5837b0e5-9eae-4ae6-8367-37c451dd6bae None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:50.918057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5837b0e5-9eae-4ae6-8367-37c451dd6bae None admin] PATH_INFO: `/v3/users/26c891a8d07c43a5bf3cae8b3acdc726` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:50.920800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5837b0e5-9eae-4ae6-8367-37c451dd6bae None admin] RBAC: Authorizing `identity:get_user(user_id=26c891a8d07c43a5bf3cae8b3acdc726)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:50.922432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5837b0e5-9eae-4ae6-8367-37c451dd6bae None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:50.924729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 77/146] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:50 2022] GET /identity/v3/users/26c891a8d07c43a5bf3cae8b3acdc726 => generated 289 bytes in 476 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:50.947124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48e909f-e15a-4149-b0cf-d857e4d08da5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:51.072294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:51.090890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48e909f-e15a-4149-b0cf-d857e4d08da5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:51.127452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9643722-0a8d-4c99-baa7-79b2b13ea84a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:51.128126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9643722-0a8d-4c99-baa7-79b2b13ea84a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:51.128411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9643722-0a8d-4c99-baa7-79b2b13ea84a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:51.128745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9643722-0a8d-4c99-baa7-79b2b13ea84a None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:51.130175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9643722-0a8d-4c99-baa7-79b2b13ea84a None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:51.131759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9643722-0a8d-4c99-baa7-79b2b13ea84a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:51.133826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 70/147] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:50 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:51.139364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5837b0e5-9eae-4ae6-8367-37c451dd6bae None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:51.273766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5837b0e5-9eae-4ae6-8367-37c451dd6bae None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:51.280698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:51.324509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e38755-efa1-4565-8ddf-368bdae5db97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:51.325140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e38755-efa1-4565-8ddf-368bdae5db97 None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:51.325424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e38755-efa1-4565-8ddf-368bdae5db97 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:51.325700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e38755-efa1-4565-8ddf-368bdae5db97 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/users/26c891a8d07c43a5bf3cae8b3acdc726/roles/9950301e6a2d4ba594bc506ec017aad0` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:51.327741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e38755-efa1-4565-8ddf-368bdae5db97 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d187412d8794cd7a7ad12d6f240c67e, user_id=26c891a8d07c43a5bf3cae8b3acdc726, role_id=9950301e6a2d4ba594bc506ec017aad0)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:51.329223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e38755-efa1-4565-8ddf-368bdae5db97 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:51.350285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 78/148] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:55:51 2022] PUT /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/users/26c891a8d07c43a5bf3cae8b3acdc726/roles/9950301e6a2d4ba594bc506ec017aad0 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:55:51.544005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3cc3b0-f392-490c-bec1-138fa336bf26 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:51.544398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3cc3b0-f392-490c-bec1-138fa336bf26 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:51.544586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3cc3b0-f392-490c-bec1-138fa336bf26 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:51.546295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 71/149] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:51.552509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf655f-634f-411a-b6aa-a8dce6279bfa None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:51.552822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf655f-634f-411a-b6aa-a8dce6279bfa None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:51.553131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf655f-634f-411a-b6aa-a8dce6279bfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:51.577991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-75cf655f-634f-411a-b6aa-a8dce6279bfa None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:51.579227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-75cf655f-634f-411a-b6aa-a8dce6279bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:51.674167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 79/150] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:51.982785 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3cc3b0-f392-490c-bec1-138fa336bf26 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:52.117370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3cc3b0-f392-490c-bec1-138fa336bf26 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:52.160422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2f25e2-eedd-4d01-8eec-98d9e1af2052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:52.161597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f25e2-eedd-4d01-8eec-98d9e1af2052 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:52.161877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f25e2-eedd-4d01-8eec-98d9e1af2052 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:52.162148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f25e2-eedd-4d01-8eec-98d9e1af2052 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:52.173244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2f25e2-eedd-4d01-8eec-98d9e1af2052 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-6b2f25e2-eedd-4d01-8eec-98d9e1af2052 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:52.181534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:52.182436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:52.183220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 72/151] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:55:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:52.188549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cf655f-634f-411a-b6aa-a8dce6279bfa None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:52.296297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:52.309135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cf655f-634f-411a-b6aa-a8dce6279bfa None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:52.360408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:52.361083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:52.361373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:52.361701 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:52.362380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:52.362739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:52.364281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:52.378547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 80/152] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:52.384837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2f25e2-eedd-4d01-8eec-98d9e1af2052 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:52.405939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:52.516827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2f25e2-eedd-4d01-8eec-98d9e1af2052 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:52.556101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88d0d09-c8b1-402b-8ba9-c9b28288d5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:52.556719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88d0d09-c8b1-402b-8ba9-c9b28288d5aa None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:52.557067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88d0d09-c8b1-402b-8ba9-c9b28288d5aa None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:52.557349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88d0d09-c8b1-402b-8ba9-c9b28288d5aa None admin] PATH_INFO: `/v3/users/22bd926197084ae38005e05ce6f936ab` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:52.558491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88d0d09-c8b1-402b-8ba9-c9b28288d5aa None admin] RBAC: Authorizing `identity:get_user(user_id=22bd926197084ae38005e05ce6f936ab)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:52.559985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88d0d09-c8b1-402b-8ba9-c9b28288d5aa None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:52.561576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 73/153] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:52 2022] GET /identity/v3/users/22bd926197084ae38005e05ce6f936ab => generated 247 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:52.567863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:52.679624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4577a700-3ea7-4b4a-a496-b5bc02abf9ca None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:52.679977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4577a700-3ea7-4b4a-a496-b5bc02abf9ca None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:52.680208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4577a700-3ea7-4b4a-a496-b5bc02abf9ca None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:52.681864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 74/154] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:52.687463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857613ca-beee-4407-a753-046526b82914 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:52.687772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857613ca-beee-4407-a753-046526b82914 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:52.688054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857613ca-beee-4407-a753-046526b82914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:52.712528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864fb2a8-cd1f-4f2a-9147-c5962183204c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:52.720692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-857613ca-beee-4407-a753-046526b82914 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:52.722365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-857613ca-beee-4407-a753-046526b82914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:52.776636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946cdffc-8383-41fa-95ca-a08027a2d19f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:52.777414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946cdffc-8383-41fa-95ca-a08027a2d19f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:52.777702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946cdffc-8383-41fa-95ca-a08027a2d19f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:52.778031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946cdffc-8383-41fa-95ca-a08027a2d19f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:52.815123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:52.844470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 75/155] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:52.881492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946cdffc-8383-41fa-95ca-a08027a2d19f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:52.883128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946cdffc-8383-41fa-95ca-a08027a2d19f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:52.975767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479bbf7b-c966-4f6b-b29c-7c146b9a40d6 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:52.976064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479bbf7b-c966-4f6b-b29c-7c146b9a40d6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:52.976336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479bbf7b-c966-4f6b-b29c-7c146b9a40d6 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:52.977817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 76/156] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:55:52.995937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7cc76-913d-407b-80ef-d36655ed1c1f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:52.997699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7cc76-913d-407b-80ef-d36655ed1c1f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:52.998316 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d7cc76-913d-407b-80ef-d36655ed1c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:53.034845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 81/157] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:53.035446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-59d7cc76-913d-407b-80ef-d36655ed1c1f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:53.037554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-59d7cc76-913d-407b-80ef-d36655ed1c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:53.053886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90854dd-588a-40e4-b25c-167dd3b1ac48 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:53.054705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90854dd-588a-40e4-b25c-167dd3b1ac48 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:53.055169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90854dd-588a-40e4-b25c-167dd3b1ac48 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:53.056799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 82/158] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:55:53.058457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90854dd-588a-40e4-b25c-167dd3b1ac48 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:53.059037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:53.178296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:53.181456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 77/159] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:55:53.183180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bdec1-7a71-4a5d-90d8-4e1549e0cf49 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:53.188411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bdec1-7a71-4a5d-90d8-4e1549e0cf49 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:53.188974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bdec1-7a71-4a5d-90d8-4e1549e0cf49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:53.236716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6b3bdec1-7a71-4a5d-90d8-4e1549e0cf49 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:53.237909 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6b3bdec1-7a71-4a5d-90d8-4e1549e0cf49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:53.268349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90854dd-588a-40e4-b25c-167dd3b1ac48 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:53.328719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464eec93-8b5c-4274-8c3d-501189e72d59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:53.330129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464eec93-8b5c-4274-8c3d-501189e72d59 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:53.330656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464eec93-8b5c-4274-8c3d-501189e72d59 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:53.331114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464eec93-8b5c-4274-8c3d-501189e72d59 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:53.336353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464eec93-8b5c-4274-8c3d-501189e72d59 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:53.338025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464eec93-8b5c-4274-8c3d-501189e72d59 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:53.340024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 83/160] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:53 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:53.346002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464eec93-8b5c-4274-8c3d-501189e72d59 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:53.374592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 78/161] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:53.432998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3bdec1-7a71-4a5d-90d8-4e1549e0cf49 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:53.503464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464eec93-8b5c-4274-8c3d-501189e72d59 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:53.544790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:53.557126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:53.559970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:53.560898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:53.561350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:53.561789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:53.562692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e, user.id=22bd926197084ae38005e05ce6f936ab) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:53.563172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:53.564833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:53.576142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 84/162] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:53 2022] GET /identity/v3/role_assignments?user.id=22bd926197084ae38005e05ce6f936ab&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 608 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:53.622780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3bdec1-7a71-4a5d-90d8-4e1549e0cf49 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:53.671990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b986a2-58db-4929-a91c-58c08c6ffce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:53.672629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b986a2-58db-4929-a91c-58c08c6ffce7 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:53.672915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b986a2-58db-4929-a91c-58c08c6ffce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:53.673177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b986a2-58db-4929-a91c-58c08c6ffce7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:53.696807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b986a2-58db-4929-a91c-58c08c6ffce7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-03b986a2-58db-4929-a91c-58c08c6ffce7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:53.701729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:53.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:53.703480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 79/163] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:53.708961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:53.854368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4b0af-8768-4689-a5aa-486d7e27d674 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:53.891524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:53.892597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:53.893087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:53.893570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:53.893924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b986a2-58db-4929-a91c-58c08c6ffce7 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:53.894780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:53.895840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:53.897452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:53.907219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 85/164] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:53.915415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:53.985580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:54.039263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d479a6-dff2-4d3e-812c-a94c906abf63 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:54.084878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162bd1e2-847d-41af-aeec-31a728241e33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:54.085906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162bd1e2-847d-41af-aeec-31a728241e33 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:54.086400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162bd1e2-847d-41af-aeec-31a728241e33 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:54.086874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162bd1e2-847d-41af-aeec-31a728241e33 None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:54.089089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162bd1e2-847d-41af-aeec-31a728241e33 None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:54.090808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162bd1e2-847d-41af-aeec-31a728241e33 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:54.092718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 86/165] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:53 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:54.093721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162bd1e2-847d-41af-aeec-31a728241e33 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:54.169853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b986a2-58db-4929-a91c-58c08c6ffce7 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:54.244834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:54.261725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f255c4-3ae9-4fcd-89a5-037f8871b793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:54.262367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f255c4-3ae9-4fcd-89a5-037f8871b793 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:54.262645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f255c4-3ae9-4fcd-89a5-037f8871b793 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:54.262916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f255c4-3ae9-4fcd-89a5-037f8871b793 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:54.294176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f255c4-3ae9-4fcd-89a5-037f8871b793 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-42f255c4-3ae9-4fcd-89a5-037f8871b793 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:54.299795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:54.301658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:54.303214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 80/166] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 408 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:54.307275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f255c4-3ae9-4fcd-89a5-037f8871b793 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:54.366194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162bd1e2-847d-41af-aeec-31a728241e33 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:54.395280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:54.440396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51af907-c867-401a-8855-1ecba5079b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:54.441048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51af907-c867-401a-8855-1ecba5079b78 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:54.441327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51af907-c867-401a-8855-1ecba5079b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:54.441597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51af907-c867-401a-8855-1ecba5079b78 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:54.458549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51af907-c867-401a-8855-1ecba5079b78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-e51af907-c867-401a-8855-1ecba5079b78 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:54.463826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:54.464851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:54.465803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 87/167] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:55:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 371 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:54.467514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51af907-c867-401a-8855-1ecba5079b78 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:54.500021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f255c4-3ae9-4fcd-89a5-037f8871b793 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:54.572916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cd7390-5c5b-4026-883a-c48ac5809f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:54.573528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd7390-5c5b-4026-883a-c48ac5809f7c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:54.573802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd7390-5c5b-4026-883a-c48ac5809f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:54.574067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd7390-5c5b-4026-883a-c48ac5809f7c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:54.604471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51af907-c867-401a-8855-1ecba5079b78 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:54.642594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:54.643762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:54.644235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:54.644696 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:54.645689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:54.646175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:54.647769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:54.657644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 88/168] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:54.658856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:54.692067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:54.710119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cd7390-5c5b-4026-883a-c48ac5809f7c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:54.711492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cd7390-5c5b-4026-883a-c48ac5809f7c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:54.712559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:54.820572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9496fd-6b4b-454e-99d3-afc0cd7b8b2b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:54.879527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 81/169] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 577 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:54.881346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cd7390-5c5b-4026-883a-c48ac5809f7c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:54.892402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:54.893038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:54.893319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:54.893589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:54.894256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:54.894583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:54.896779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:54.912476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 89/170] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:55:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:54.914363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:54.983304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:54.994607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cd7390-5c5b-4026-883a-c48ac5809f7c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:55.032842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738ce823-95ee-4e29-9c8f-4ec2566fc692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:55.033904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ce823-95ee-4e29-9c8f-4ec2566fc692 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.034376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ce823-95ee-4e29-9c8f-4ec2566fc692 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.034911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738ce823-95ee-4e29-9c8f-4ec2566fc692 None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.036457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738ce823-95ee-4e29-9c8f-4ec2566fc692 None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:55.038205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738ce823-95ee-4e29-9c8f-4ec2566fc692 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:55.039982 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 82/171] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:54 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:55.041018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738ce823-95ee-4e29-9c8f-4ec2566fc692 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:55.158491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff827c-2a34-4a77-88ff-c5d1dca43f32 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:55.160144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:55.163135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738ce823-95ee-4e29-9c8f-4ec2566fc692 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:55.203969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f6c4f-712b-4f37-953e-dfa926cc4cda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:55.205476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f6c4f-712b-4f37-953e-dfa926cc4cda None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.205955 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f6c4f-712b-4f37-953e-dfa926cc4cda None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.206405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f6c4f-712b-4f37-953e-dfa926cc4cda None admin] PATH_INFO: `/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.207922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0f6c4f-712b-4f37-953e-dfa926cc4cda None admin] RBAC: Authorizing `identity:get_user(user_id=054bf47407b34e4a875fbfb8ad1c5a1b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:55.209840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0f6c4f-712b-4f37-953e-dfa926cc4cda None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:55.212168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 83/172] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:55 2022] GET /identity/v3/users/054bf47407b34e4a875fbfb8ad1c5a1b => generated 275 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:55.213477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f6c4f-712b-4f37-953e-dfa926cc4cda None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:55.231997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571c267-0b4f-4033-b8d5-b44875c1c483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:55.232652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571c267-0b4f-4033-b8d5-b44875c1c483 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.232920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571c267-0b4f-4033-b8d5-b44875c1c483 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.233205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571c267-0b4f-4033-b8d5-b44875c1c483 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.235365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f571c267-0b4f-4033-b8d5-b44875c1c483 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:55.237006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f571c267-0b4f-4033-b8d5-b44875c1c483 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:55.239298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 90/173] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:54 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:55.240126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571c267-0b4f-4033-b8d5-b44875c1c483 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:55.248361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:55.326608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f6c4f-712b-4f37-953e-dfa926cc4cda None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:55.363162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a23f904-91fe-44a0-bdce-1dc12afdd635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:55.364357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23f904-91fe-44a0-bdce-1dc12afdd635 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.364845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23f904-91fe-44a0-bdce-1dc12afdd635 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.365315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23f904-91fe-44a0-bdce-1dc12afdd635 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.498207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571c267-0b4f-4033-b8d5-b44875c1c483 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:55.517282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a23f904-91fe-44a0-bdce-1dc12afdd635 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:55.518891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a23f904-91fe-44a0-bdce-1dc12afdd635 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:55.536566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679019b-abdd-4fe7-8cab-a5db61a271fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:55.537930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679019b-abdd-4fe7-8cab-a5db61a271fd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.538385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679019b-abdd-4fe7-8cab-a5db61a271fd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.538870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679019b-abdd-4fe7-8cab-a5db61a271fd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.588895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:55.590069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:55.621773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c679019b-abdd-4fe7-8cab-a5db61a271fd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:55.623707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c679019b-abdd-4fe7-8cab-a5db61a271fd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:55.734940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 91/174] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:55.736760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cca49f-0443-4577-bb1f-6a27b91f712b None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.737188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cca49f-0443-4577-bb1f-6a27b91f712b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.737493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cca49f-0443-4577-bb1f-6a27b91f712b None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.739393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 92/175] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:55.740276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cca49f-0443-4577-bb1f-6a27b91f712b None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:55.741666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 84/176] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 529 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:55.743177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ef35e-cd05-4fd2-8225-36f419623019 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.743725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ef35e-cd05-4fd2-8225-36f419623019 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.744258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ef35e-cd05-4fd2-8225-36f419623019 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.745736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 85/177] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:55.751247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91263b5-8988-4a05-aead-59d6ddf3b031 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.751912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91263b5-8988-4a05-aead-59d6ddf3b031 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.752380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91263b5-8988-4a05-aead-59d6ddf3b031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.772635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b91263b5-8988-4a05-aead-59d6ddf3b031 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:55.774264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b91263b5-8988-4a05-aead-59d6ddf3b031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:55.815986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:55.849815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 86/178] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:55.850733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cca49f-0443-4577-bb1f-6a27b91f712b None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:55.851221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91263b5-8988-4a05-aead-59d6ddf3b031 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:55.888819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb88e1a1-1594-4017-b8f8-59facac44cf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:55.889522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb88e1a1-1594-4017-b8f8-59facac44cf8 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.890004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb88e1a1-1594-4017-b8f8-59facac44cf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.890480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb88e1a1-1594-4017-b8f8-59facac44cf8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.891852 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb88e1a1-1594-4017-b8f8-59facac44cf8 None admin] RBAC: Adding query filter params (role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e, user.id=054bf47407b34e4a875fbfb8ad1c5a1b) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:55.892418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb88e1a1-1594-4017-b8f8-59facac44cf8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:55.895614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb88e1a1-1594-4017-b8f8-59facac44cf8 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:55.903901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 93/179] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:55 2022] GET /identity/v3/role_assignments?user.id=054bf47407b34e4a875fbfb8ad1c5a1b&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:55.905284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251dad6-6a0b-499c-95ed-bc6debc894d8 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:55.905569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251dad6-6a0b-499c-95ed-bc6debc894d8 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:55.905928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251dad6-6a0b-499c-95ed-bc6debc894d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:55.916718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:55.930000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-0251dad6-6a0b-499c-95ed-bc6debc894d8 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:55.931193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0251dad6-6a0b-499c-95ed-bc6debc894d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:55.968095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91263b5-8988-4a05-aead-59d6ddf3b031 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:56.007341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b726c5-0a1e-4fab-834f-3bd00feef442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:56.008449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b726c5-0a1e-4fab-834f-3bd00feef442 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.008984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b726c5-0a1e-4fab-834f-3bd00feef442 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.009450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b726c5-0a1e-4fab-834f-3bd00feef442 None admin] PATH_INFO: `/v3/projects/65e73a0199b148dea188f22c07251e3c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.010986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b726c5-0a1e-4fab-834f-3bd00feef442 None admin] RBAC: Authorizing `identity:get_project(project_id=65e73a0199b148dea188f22c07251e3c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:56.012753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b726c5-0a1e-4fab-834f-3bd00feef442 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:56.014807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 87/180] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:55 2022] GET /identity/v3/projects/65e73a0199b148dea188f22c07251e3c => generated 312 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:56.015904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b726c5-0a1e-4fab-834f-3bd00feef442 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:56.016699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 94/181] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:56.023727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251dad6-6a0b-499c-95ed-bc6debc894d8 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:56.132072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:56.182625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b726c5-0a1e-4fab-834f-3bd00feef442 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:56.207157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251dad6-6a0b-499c-95ed-bc6debc894d8 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:56.229686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:56.253309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db14a1e-86a2-44a1-a15a-0a73368c73a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:56.253969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db14a1e-86a2-44a1-a15a-0a73368c73a2 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.254359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db14a1e-86a2-44a1-a15a-0a73368c73a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.254716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db14a1e-86a2-44a1-a15a-0a73368c73a2 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.255986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db14a1e-86a2-44a1-a15a-0a73368c73a2 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:56.257809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db14a1e-86a2-44a1-a15a-0a73368c73a2 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:56.259560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 88/182] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:56.268238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d83dec-7a58-4529-b53a-853486bc48a2 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.268558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d83dec-7a58-4529-b53a-853486bc48a2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.268807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d83dec-7a58-4529-b53a-853486bc48a2 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.271158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 89/183] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:56 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:56.271984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d83dec-7a58-4529-b53a-853486bc48a2 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:56.304474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26604c7-b8ff-42a4-a220-d08516d008c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:56.305170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26604c7-b8ff-42a4-a220-d08516d008c2 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.307727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26604c7-b8ff-42a4-a220-d08516d008c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.307727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26604c7-b8ff-42a4-a220-d08516d008c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.307727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26604c7-b8ff-42a4-a220-d08516d008c2 None admin] RBAC: Adding query filter params (role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=65e73a0199b148dea188f22c07251e3c, user.id=054bf47407b34e4a875fbfb8ad1c5a1b) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:56.307727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26604c7-b8ff-42a4-a220-d08516d008c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:56.308223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26604c7-b8ff-42a4-a220-d08516d008c2 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:56.319133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 95/184] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/role_assignments?user.id=054bf47407b34e4a875fbfb8ad1c5a1b&scope.project.id=65e73a0199b148dea188f22c07251e3c&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 608 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:55:56.320598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4692f6-2b13-4c9e-b507-3c1949cfd1f7 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.324143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4692f6-2b13-4c9e-b507-3c1949cfd1f7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.324411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4692f6-2b13-4c9e-b507-3c1949cfd1f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.355951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9b4692f6-2b13-4c9e-b507-3c1949cfd1f7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:56.357129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9b4692f6-2b13-4c9e-b507-3c1949cfd1f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:56.483207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d83dec-7a58-4529-b53a-853486bc48a2 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:56.483920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 96/185] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:56.485359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91265d2a-1c96-450c-9f9b-a105356a3a4f None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.485643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91265d2a-1c96-450c-9f9b-a105356a3a4f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.485973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91265d2a-1c96-450c-9f9b-a105356a3a4f None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.487497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 97/186] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:55:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:55:56.491513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91265d2a-1c96-450c-9f9b-a105356a3a4f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:56.532412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:56.533484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.533960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.534389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.535273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] RBAC: Adding query filter params (role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e, user.id=054bf47407b34e4a875fbfb8ad1c5a1b) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:56.536725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:56.538521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:56.545993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 90/187] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/role_assignments?user.id=054bf47407b34e4a875fbfb8ad1c5a1b&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 608 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:56.547069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:56.573664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:56.623672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:56.628975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91265d2a-1c96-450c-9f9b-a105356a3a4f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:56.671753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871b5186-53ab-4e2d-ac49-7fb37912280e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:56.672423 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871b5186-53ab-4e2d-ac49-7fb37912280e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.672740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871b5186-53ab-4e2d-ac49-7fb37912280e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.673023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871b5186-53ab-4e2d-ac49-7fb37912280e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.678427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d761a5e9-7a87-43b9-b46f-751a70759fff None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:56.687075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871b5186-53ab-4e2d-ac49-7fb37912280e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-871b5186-53ab-4e2d-ac49-7fb37912280e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:56.693240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:56.694205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:55:56.694995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 98/188] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:56.695940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0776fdf-38ec-436a-986b-8bf2d158f101 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.696070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0776fdf-38ec-436a-986b-8bf2d158f101 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.696347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0776fdf-38ec-436a-986b-8bf2d158f101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.721471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e847a0d-33b3-41a1-96a8-22302b481649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:56.722368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e847a0d-33b3-41a1-96a8-22302b481649 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.722628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d0776fdf-38ec-436a-986b-8bf2d158f101 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:55:56.723531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e847a0d-33b3-41a1-96a8-22302b481649 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.724217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e847a0d-33b3-41a1-96a8-22302b481649 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.724498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d0776fdf-38ec-436a-986b-8bf2d158f101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:55:56.737850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e847a0d-33b3-41a1-96a8-22302b481649 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-7e847a0d-33b3-41a1-96a8-22302b481649 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:56.745213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:56.746310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:55:56.747226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:56.749209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 91/189] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:56.750140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e847a0d-33b3-41a1-96a8-22302b481649 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:56.811924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 99/190] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:55:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:55:56.812612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0776fdf-38ec-436a-986b-8bf2d158f101 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:56.872115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:56.880912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e847a0d-33b3-41a1-96a8-22302b481649 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:56.902206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:56.922436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:56.923047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.923324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.923598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.924285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:56.924583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:56.926330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:56.933154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0776fdf-38ec-436a-986b-8bf2d158f101 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:56.934588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 92/191] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:56.935448 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:56.971794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:56.972404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:56.972680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:56.973075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:56.973754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:56.974058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:56.975434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:56.983894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 100/192] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:56.984819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.043359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f224e8-1bee-4c13-9e97-945907c3c2f0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.079144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1f1cd9-0514-41e0-8ed4-4ab763ad70a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.079714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1f1cd9-0514-41e0-8ed4-4ab763ad70a4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.080021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1f1cd9-0514-41e0-8ed4-4ab763ad70a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.080294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1f1cd9-0514-41e0-8ed4-4ab763ad70a4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.089749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7712808-0ae0-4aaa-b7ec-06e4ff2eb027 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.091002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1f1cd9-0514-41e0-8ed4-4ab763ad70a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-af1f1cd9-0514-41e0-8ed4-4ab763ad70a4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:55:57.096265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:55:57.097221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 93/193] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:55:57.102901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1f1cd9-0514-41e0-8ed4-4ab763ad70a4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.107496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:57.134971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592291ce-615b-42fb-9716-a45ef63d6d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.135601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592291ce-615b-42fb-9716-a45ef63d6d24 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.135904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592291ce-615b-42fb-9716-a45ef63d6d24 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.136177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592291ce-615b-42fb-9716-a45ef63d6d24 None admin] PATH_INFO: `/v3/users/26c891a8d07c43a5bf3cae8b3acdc726` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.137324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592291ce-615b-42fb-9716-a45ef63d6d24 None admin] RBAC: Authorizing `identity:get_user(user_id=26c891a8d07c43a5bf3cae8b3acdc726)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.138779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592291ce-615b-42fb-9716-a45ef63d6d24 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.140385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 101/194] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:56 2022] GET /identity/v3/users/26c891a8d07c43a5bf3cae8b3acdc726 => generated 289 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:57.141142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592291ce-615b-42fb-9716-a45ef63d6d24 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.143970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:57.204427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1f1cd9-0514-41e0-8ed4-4ab763ad70a4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.239568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c25c4d-65f5-4357-a93b-bf1a27126c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.240235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c25c4d-65f5-4357-a93b-bf1a27126c1d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.240521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c25c4d-65f5-4357-a93b-bf1a27126c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.240797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c25c4d-65f5-4357-a93b-bf1a27126c1d None admin] PATH_INFO: `/v3/users/4c6fa3433f69465187738d533e6044a2` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.243919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592291ce-615b-42fb-9716-a45ef63d6d24 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.253664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c25c4d-65f5-4357-a93b-bf1a27126c1d None admin] RBAC: Authorizing `identity:get_user(user_id=4c6fa3433f69465187738d533e6044a2)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.255158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c25c4d-65f5-4357-a93b-bf1a27126c1d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.256806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 94/195] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/users/4c6fa3433f69465187738d533e6044a2 => generated 283 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:57.257568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c25c4d-65f5-4357-a93b-bf1a27126c1d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.277650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.278255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.278535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.278807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.279469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:57.279800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.281184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.289092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 102/196] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:57.293312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.336780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:57.352382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:57.358671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c25c4d-65f5-4357-a93b-bf1a27126c1d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.391926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce320802-8f34-42b3-9bf2-478becae315b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.392549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce320802-8f34-42b3-9bf2-478becae315b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.392840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce320802-8f34-42b3-9bf2-478becae315b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.393100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce320802-8f34-42b3-9bf2-478becae315b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.393977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec512a47-9de9-483a-87f4-d459731e00c3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.394196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce320802-8f34-42b3-9bf2-478becae315b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.395720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce320802-8f34-42b3-9bf2-478becae315b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.397277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 95/197] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:57.398037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce320802-8f34-42b3-9bf2-478becae315b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.427568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89d8c4a-8705-4c17-b71d-127a523287c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.428208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d8c4a-8705-4c17-b71d-127a523287c5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.428493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d8c4a-8705-4c17-b71d-127a523287c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.428767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d8c4a-8705-4c17-b71d-127a523287c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.494887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89d8c4a-8705-4c17-b71d-127a523287c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.496101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce320802-8f34-42b3-9bf2-478becae315b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.496380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89d8c4a-8705-4c17-b71d-127a523287c5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.529188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0fd590-746c-4a55-a9aa-971b3a693835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.529792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0fd590-746c-4a55-a9aa-971b3a693835 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.530078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0fd590-746c-4a55-a9aa-971b3a693835 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.530358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0fd590-746c-4a55-a9aa-971b3a693835 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.536764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:57.562267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:57.594723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 103/198] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:57.595480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89d8c4a-8705-4c17-b71d-127a523287c5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.602069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0fd590-746c-4a55-a9aa-971b3a693835 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.603513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0fd590-746c-4a55-a9aa-971b3a693835 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.693396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89d8c4a-8705-4c17-b71d-127a523287c5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.703240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 96/199] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:57.704079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0fd590-746c-4a55-a9aa-971b3a693835 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.731377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd0560d-5b95-4729-9a9a-70aa4cf5276c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.732031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0560d-5b95-4729-9a9a-70aa4cf5276c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.732307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0560d-5b95-4729-9a9a-70aa4cf5276c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.732576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0560d-5b95-4729-9a9a-70aa4cf5276c None admin] PATH_INFO: `/v3/users/4c6fa3433f69465187738d533e6044a2` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.733678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd0560d-5b95-4729-9a9a-70aa4cf5276c None admin] RBAC: Authorizing `identity:get_user(user_id=4c6fa3433f69465187738d533e6044a2)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.735116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd0560d-5b95-4729-9a9a-70aa4cf5276c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.736703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 104/200] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/users/4c6fa3433f69465187738d533e6044a2 => generated 283 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:57.737444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd0560d-5b95-4729-9a9a-70aa4cf5276c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.749308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:57.770037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:57.807280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0fd590-746c-4a55-a9aa-971b3a693835 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.836303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd0560d-5b95-4729-9a9a-70aa4cf5276c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.841019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81467f91-1386-463f-b14d-bac4debbc7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.841623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81467f91-1386-463f-b14d-bac4debbc7a8 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.841901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81467f91-1386-463f-b14d-bac4debbc7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.842171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81467f91-1386-463f-b14d-bac4debbc7a8 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.842811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81467f91-1386-463f-b14d-bac4debbc7a8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.844277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81467f91-1386-463f-b14d-bac4debbc7a8 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.865149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 97/201] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:55:57 2022] POST /identity/v3/users => generated 297 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:55:57.865895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81467f91-1386-463f-b14d-bac4debbc7a8 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.888213 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff010c3-761c-4beb-8f47-972e182ea3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:57.889471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff010c3-761c-4beb-8f47-972e182ea3db None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:57.889841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff010c3-761c-4beb-8f47-972e182ea3db None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:57.890126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff010c3-761c-4beb-8f47-972e182ea3db None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:57.891366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff010c3-761c-4beb-8f47-972e182ea3db None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:57.893053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff010c3-761c-4beb-8f47-972e182ea3db None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:57.895385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 105/202] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:57.901274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff010c3-761c-4beb-8f47-972e182ea3db None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:57.994506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81467f91-1386-463f-b14d-bac4debbc7a8 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:57.995955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:58.018341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:58.025514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff010c3-761c-4beb-8f47-972e182ea3db None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:58.034830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb3a43c-4411-435d-8c06-cc41d852e528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:58.035621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb3a43c-4411-435d-8c06-cc41d852e528 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:58.035971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb3a43c-4411-435d-8c06-cc41d852e528 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:58.036258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb3a43c-4411-435d-8c06-cc41d852e528 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:58.037409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb3a43c-4411-435d-8c06-cc41d852e528 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:58.038878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb3a43c-4411-435d-8c06-cc41d852e528 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:58.040569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 98/203] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:58.041338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb3a43c-4411-435d-8c06-cc41d852e528 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:58.073240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062a086-a9c1-4e73-9506-78a270752b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:58.074125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062a086-a9c1-4e73-9506-78a270752b1d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:58.074405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062a086-a9c1-4e73-9506-78a270752b1d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:58.074826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9062a086-a9c1-4e73-9506-78a270752b1d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:58.157703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9062a086-a9c1-4e73-9506-78a270752b1d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:58.159390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9062a086-a9c1-4e73-9506-78a270752b1d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:58.172557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb3a43c-4411-435d-8c06-cc41d852e528 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:58.210806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:58.212489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:58.212489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:58.212648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:58.213318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] RBAC: Adding query filter params (role.id=9950301e6a2d4ba594bc506ec017aad0, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e, user.id=26c891a8d07c43a5bf3cae8b3acdc726) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:58.213810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:58.215659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:58.224908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 99/204] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:58 2022] GET /identity/v3/role_assignments?user.id=26c891a8d07c43a5bf3cae8b3acdc726&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=9950301e6a2d4ba594bc506ec017aad0 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:58.225963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:58.253135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:58.294628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 106/205] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:55:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:55:58.303305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062a086-a9c1-4e73-9506-78a270752b1d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:58.314531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:58.369433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2cfa46-890e-4a51-9dd0-508873d7b913 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:58.407261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:58.408251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:58.408672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:58.409129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:58.410001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7, user.id=4c6fa3433f69465187738d533e6044a2) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:58.410991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:58.413091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:58.419738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 100/206] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:58 2022] GET /identity/v3/role_assignments?user.id=4c6fa3433f69465187738d533e6044a2&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:58.434100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9062a086-a9c1-4e73-9506-78a270752b1d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:58.496217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8be160-34c1-4a47-9b5b-f69c998cc9b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:58.496876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8be160-34c1-4a47-9b5b-f69c998cc9b4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:58.497158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8be160-34c1-4a47-9b5b-f69c998cc9b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:58.497433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8be160-34c1-4a47-9b5b-f69c998cc9b4 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:58.499534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8be160-34c1-4a47-9b5b-f69c998cc9b4 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:58.501061 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8be160-34c1-4a47-9b5b-f69c998cc9b4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:58.502816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 107/207] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:55:58 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:55:58.509367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:55:58.632981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee4dca3-5b79-4ffc-b61a-deeb8dabc656 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:55:58.651201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:55:58.673940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e236546-4544-4e93-87c3-349773414997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:55:58.674899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e236546-4544-4e93-87c3-349773414997 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:55:58.675377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e236546-4544-4e93-87c3-349773414997 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:55:58.676682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e236546-4544-4e93-87c3-349773414997 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:55:58.678080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e236546-4544-4e93-87c3-349773414997 None admin] RBAC: Adding query filter params (role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7, user.id=4c6fa3433f69465187738d533e6044a2) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:55:58.678590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e236546-4544-4e93-87c3-349773414997 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:55:58.680267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e236546-4544-4e93-87c3-349773414997 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:55:58.687923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 101/208] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:55:58 2022] GET /identity/v3/role_assignments?user.id=4c6fa3433f69465187738d533e6044a2&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:00.305331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e70a55-174e-48fe-a6f0-ed8e0e8f9190 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.305838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e70a55-174e-48fe-a6f0-ed8e0e8f9190 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.305942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e70a55-174e-48fe-a6f0-ed8e0e8f9190 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.307228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 108/209] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:00.316576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc67941-afc7-42f1-9e06-c37f0b86491f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.317351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc67941-afc7-42f1-9e06-c37f0b86491f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.317823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc67941-afc7-42f1-9e06-c37f0b86491f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.336778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-3bc67941-afc7-42f1-9e06-c37f0b86491f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:00.338258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3bc67941-afc7-42f1-9e06-c37f0b86491f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:00.397384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579ab734-0e72-44aa-8c27-b7f6684d3302 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.397997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579ab734-0e72-44aa-8c27-b7f6684d3302 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.398489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579ab734-0e72-44aa-8c27-b7f6684d3302 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.400319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 109/210] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:00.405826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568960e-a922-433d-9f1d-ddab2f39ad1c None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.406345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568960e-a922-433d-9f1d-ddab2f39ad1c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.406827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568960e-a922-433d-9f1d-ddab2f39ad1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.412567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 102/211] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:00.427289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-b568960e-a922-433d-9f1d-ddab2f39ad1c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:00.428804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b568960e-a922-433d-9f1d-ddab2f39ad1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:00.471357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:00.506231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 110/212] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:00.672831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab165bc2-63b0-430d-9709-7c497743bd6e None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.673504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab165bc2-63b0-430d-9709-7c497743bd6e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.674017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab165bc2-63b0-430d-9709-7c497743bd6e None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.676123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 103/213] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:00.681369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9012eed2-6512-4a0b-b0f1-2069652de768 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.681882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9012eed2-6512-4a0b-b0f1-2069652de768 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.682358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9012eed2-6512-4a0b-b0f1-2069652de768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.700322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab165bc2-63b0-430d-9709-7c497743bd6e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:00.701837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9012eed2-6512-4a0b-b0f1-2069652de768 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:00.703024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9012eed2-6512-4a0b-b0f1-2069652de768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:00.781964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 111/214] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:00.783127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9012eed2-6512-4a0b-b0f1-2069652de768 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:00.826350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab165bc2-63b0-430d-9709-7c497743bd6e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:00.841939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:00.875835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e32a472-d820-45d7-a168-3c61dadc6b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:00.877435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32a472-d820-45d7-a168-3c61dadc6b06 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.877914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32a472-d820-45d7-a168-3c61dadc6b06 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.878410 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e32a472-d820-45d7-a168-3c61dadc6b06 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.893138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9012eed2-6512-4a0b-b0f1-2069652de768 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:00.894224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e32a472-d820-45d7-a168-3c61dadc6b06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:00.896990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-2e32a472-d820-45d7-a168-3c61dadc6b06 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:00.904511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:00.905458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:00.906235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:00.909640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 104/215] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:00.915267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e32a472-d820-45d7-a168-3c61dadc6b06 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:00.940608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c234d7f-9a79-4d27-96e2-819e2693bb88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:00.941172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c234d7f-9a79-4d27-96e2-819e2693bb88 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.941541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c234d7f-9a79-4d27-96e2-819e2693bb88 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.942328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c234d7f-9a79-4d27-96e2-819e2693bb88 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.943443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c234d7f-9a79-4d27-96e2-819e2693bb88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:00.944934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c234d7f-9a79-4d27-96e2-819e2693bb88 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:00.946760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 112/216] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:00.947777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d41fd-9dc6-477b-98aa-d1527633f862 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:00.948052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d41fd-9dc6-477b-98aa-d1527633f862 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:00.948321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7d41fd-9dc6-477b-98aa-d1527633f862 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:00.951858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 113/217] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:00.951858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7d41fd-9dc6-477b-98aa-d1527633f862 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.032829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e32a472-d820-45d7-a168-3c61dadc6b06 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.063282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7d41fd-9dc6-477b-98aa-d1527633f862 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.070104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7d26bd-74d2-4461-8029-13f75638a38b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.070717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7d26bd-74d2-4461-8029-13f75638a38b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.071002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7d26bd-74d2-4461-8029-13f75638a38b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.071268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7d26bd-74d2-4461-8029-13f75638a38b None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.071961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7d26bd-74d2-4461-8029-13f75638a38b None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:01.072264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7d26bd-74d2-4461-8029-13f75638a38b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:01.073931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7d26bd-74d2-4461-8029-13f75638a38b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:01.083021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 105/218] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:01.084411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a167d49a-f5bf-44ac-b453-86122f35f78e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.084702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a167d49a-f5bf-44ac-b453-86122f35f78e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.084976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a167d49a-f5bf-44ac-b453-86122f35f78e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.102316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b9d78a-1702-4521-9bb5-1e334320bf1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.102923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b9d78a-1702-4521-9bb5-1e334320bf1e None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.103197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b9d78a-1702-4521-9bb5-1e334320bf1e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.103466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b9d78a-1702-4521-9bb5-1e334320bf1e None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.103632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-a167d49a-f5bf-44ac-b453-86122f35f78e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:01.104143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b9d78a-1702-4521-9bb5-1e334320bf1e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:01.104838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a167d49a-f5bf-44ac-b453-86122f35f78e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:01.105580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b9d78a-1702-4521-9bb5-1e334320bf1e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:01.132768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 114/219] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:56:00 2022] POST /identity/v3/users => generated 297 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:56:01.133581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b9d78a-1702-4521-9bb5-1e334320bf1e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.134439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:01.175404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:01.187009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 106/220] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:01.188176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a167d49a-f5bf-44ac-b453-86122f35f78e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.290492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b9d78a-1702-4521-9bb5-1e334320bf1e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.323033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a167d49a-f5bf-44ac-b453-86122f35f78e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.346098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74576acc-79b3-40d1-9e35-c0ce7520b0cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.346745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74576acc-79b3-40d1-9e35-c0ce7520b0cd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.347043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74576acc-79b3-40d1-9e35-c0ce7520b0cd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.347300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74576acc-79b3-40d1-9e35-c0ce7520b0cd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.359674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74576acc-79b3-40d1-9e35-c0ce7520b0cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-74576acc-79b3-40d1-9e35-c0ce7520b0cd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:01.365543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:01.366437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:01.367220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 115/221] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:01.375596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74576acc-79b3-40d1-9e35-c0ce7520b0cd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.378459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc032631-6a80-425a-91c0-28ad152ee784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.380749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc032631-6a80-425a-91c0-28ad152ee784 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.381295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc032631-6a80-425a-91c0-28ad152ee784 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.381584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc032631-6a80-425a-91c0-28ad152ee784 None admin] PATH_INFO: `/v3/users/628f692df03d49a8a23f574d27e02b09` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.405480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc032631-6a80-425a-91c0-28ad152ee784 None admin] RBAC: Authorizing `identity:get_user(user_id=628f692df03d49a8a23f574d27e02b09)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:01.407539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc032631-6a80-425a-91c0-28ad152ee784 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:01.409454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 107/222] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/users/628f692df03d49a8a23f574d27e02b09 => generated 297 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:01.415537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc032631-6a80-425a-91c0-28ad152ee784 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.439031 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:01.493042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:01.509523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74576acc-79b3-40d1-9e35-c0ce7520b0cd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.547229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc032631-6a80-425a-91c0-28ad152ee784 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.548694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.549307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.550977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.550977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.550977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:01.551194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:01.552536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:01.567560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 116/223] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:01.567560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.595583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0467992-a8c6-4139-add7-6b57ca08746d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.596372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0467992-a8c6-4139-add7-6b57ca08746d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.596651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0467992-a8c6-4139-add7-6b57ca08746d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.596921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0467992-a8c6-4139-add7-6b57ca08746d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.674744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0467992-a8c6-4139-add7-6b57ca08746d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:01.676546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0467992-a8c6-4139-add7-6b57ca08746d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:01.694931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e51efa3-ff80-48f0-ac30-1f765bdd3a6b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.699934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:01.740330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f34aafb-8982-4a71-ab1f-e741cbdaa92a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.741309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34aafb-8982-4a71-ab1f-e741cbdaa92a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.741876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34aafb-8982-4a71-ab1f-e741cbdaa92a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.742160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34aafb-8982-4a71-ab1f-e741cbdaa92a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.760315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f34aafb-8982-4a71-ab1f-e741cbdaa92a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-8f34aafb-8982-4a71-ab1f-e741cbdaa92a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:01.765704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:01.766574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:01.767371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 117/224] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:01.773216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f34aafb-8982-4a71-ab1f-e741cbdaa92a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.787983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:01.809321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 108/225] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:56:01.810530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0467992-a8c6-4139-add7-6b57ca08746d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.903652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f34aafb-8982-4a71-ab1f-e741cbdaa92a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.923741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0467992-a8c6-4139-add7-6b57ca08746d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:01.948292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:01.960063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824ae545-76af-4cc8-90c9-52a68c9b402c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.960946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824ae545-76af-4cc8-90c9-52a68c9b402c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.961234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824ae545-76af-4cc8-90c9-52a68c9b402c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.961509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824ae545-76af-4cc8-90c9-52a68c9b402c None admin] PATH_INFO: `/v3/users/4c6fa3433f69465187738d533e6044a2` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.963190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824ae545-76af-4cc8-90c9-52a68c9b402c None admin] RBAC: Authorizing `identity:get_user(user_id=4c6fa3433f69465187738d533e6044a2)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:01.964882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824ae545-76af-4cc8-90c9-52a68c9b402c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:01.966479 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 118/226] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/users/4c6fa3433f69465187738d533e6044a2 => generated 283 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:01.967875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:01.968712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:01.969191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:01.969678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:01.970542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:01.971043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:01.971671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824ae545-76af-4cc8-90c9-52a68c9b402c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:01.973150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:01.983060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 109/227] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:01.984274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:02.087952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:02.102701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bd5a96-a4a0-4c7e-b32d-c46da2ca8ae7 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:02.118727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824ae545-76af-4cc8-90c9-52a68c9b402c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:02.155206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a8aeee-a4de-4c54-a5f3-dfb99ecc8c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:02.156236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8aeee-a4de-4c54-a5f3-dfb99ecc8c44 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:02.156726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8aeee-a4de-4c54-a5f3-dfb99ecc8c44 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:02.157201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8aeee-a4de-4c54-a5f3-dfb99ecc8c44 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:02.157971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba6acdf-b42b-43b2-a242-53d89e2310c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:02.158520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8aeee-a4de-4c54-a5f3-dfb99ecc8c44 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:02.158945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba6acdf-b42b-43b2-a242-53d89e2310c6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:02.159371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba6acdf-b42b-43b2-a242-53d89e2310c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:02.159829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba6acdf-b42b-43b2-a242-53d89e2310c6 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:02.160987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8aeee-a4de-4c54-a5f3-dfb99ecc8c44 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:02.161739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba6acdf-b42b-43b2-a242-53d89e2310c6 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:02.167542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 119/228] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:02.167542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a8aeee-a4de-4c54-a5f3-dfb99ecc8c44 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:02.168724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba6acdf-b42b-43b2-a242-53d89e2310c6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:02.171619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 110/229] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:01 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:02.180117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba6acdf-b42b-43b2-a242-53d89e2310c6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:02.235492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:02.281946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a8aeee-a4de-4c54-a5f3-dfb99ecc8c44 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:02.289115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba6acdf-b42b-43b2-a242-53d89e2310c6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:02.322379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca65b413-5b9f-486f-a694-031bd121abc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:02.323569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca65b413-5b9f-486f-a694-031bd121abc6 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:02.324170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca65b413-5b9f-486f-a694-031bd121abc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:02.324597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca65b413-5b9f-486f-a694-031bd121abc6 None admin] PATH_INFO: `/v3/users/4c6fa3433f69465187738d533e6044a2` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:02.325487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:02.326281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:02.326660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:02.326908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca65b413-5b9f-486f-a694-031bd121abc6 None admin] RBAC: Authorizing `identity:get_user(user_id=4c6fa3433f69465187738d533e6044a2)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:02.327120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:02.328107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] RBAC: Adding query filter params (role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7, user.id=628f692df03d49a8a23f574d27e02b09) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:02.328516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:02.329173 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca65b413-5b9f-486f-a694-031bd121abc6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:02.330139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:02.331286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 120/230] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:02 2022] GET /identity/v3/users/4c6fa3433f69465187738d533e6044a2 => generated 283 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:02.333974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca65b413-5b9f-486f-a694-031bd121abc6 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:02.339835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 111/231] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:02 2022] GET /identity/v3/role_assignments?user.id=628f692df03d49a8a23f574d27e02b09&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:02.341002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:02.375555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:02.544856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca65b413-5b9f-486f-a694-031bd121abc6 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:02.564338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18883469-32d9-4ff3-9e9e-671506078a6c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:02.579854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:02.589155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c94c9e1-b03a-4754-9e19-5b139e4396eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:02.589829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c94c9e1-b03a-4754-9e19-5b139e4396eb None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:02.590157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c94c9e1-b03a-4754-9e19-5b139e4396eb None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:02.590377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c94c9e1-b03a-4754-9e19-5b139e4396eb None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/users/4c6fa3433f69465187738d533e6044a2/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:02.592973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c94c9e1-b03a-4754-9e19-5b139e4396eb None admin] RBAC: Authorizing `identity:create_grant(project_id=6fa314f8793248fb8bbde6daa676e4e7, user_id=4c6fa3433f69465187738d533e6044a2, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:02.595465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c94c9e1-b03a-4754-9e19-5b139e4396eb None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:02.623937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 121/232] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:02 2022] PUT /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/users/4c6fa3433f69465187738d533e6044a2/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:02.628520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0805eed-da76-4171-8a3c-6a0dd20e5945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:02.629313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0805eed-da76-4171-8a3c-6a0dd20e5945 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:02.629601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0805eed-da76-4171-8a3c-6a0dd20e5945 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:02.629874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0805eed-da76-4171-8a3c-6a0dd20e5945 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:02.636702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0805eed-da76-4171-8a3c-6a0dd20e5945 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:02.638215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0805eed-da76-4171-8a3c-6a0dd20e5945 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:02.640147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 112/233] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:02 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:02.646458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c94c9e1-b03a-4754-9e19-5b139e4396eb None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:02.764464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:02.789012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c94c9e1-b03a-4754-9e19-5b139e4396eb None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:02.829997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab5a40f-eefc-4831-8097-c51b654c018a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:02.830674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab5a40f-eefc-4831-8097-c51b654c018a None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:02.830954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab5a40f-eefc-4831-8097-c51b654c018a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:02.831231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab5a40f-eefc-4831-8097-c51b654c018a None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/users/4c6fa3433f69465187738d533e6044a2/roles/a0256b6f3d6c4517be88b324fcbc1145` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:02.834347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab5a40f-eefc-4831-8097-c51b654c018a None admin] RBAC: Authorizing `identity:create_grant(project_id=6fa314f8793248fb8bbde6daa676e4e7, user_id=4c6fa3433f69465187738d533e6044a2, role_id=a0256b6f3d6c4517be88b324fcbc1145)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:02.835840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab5a40f-eefc-4831-8097-c51b654c018a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:02.851908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 122/234] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:02 2022] PUT /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/users/4c6fa3433f69465187738d533e6044a2/roles/a0256b6f3d6c4517be88b324fcbc1145 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:03.597344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e09fc-1700-4fa0-9244-d4a4a97a852c None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:03.598936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e09fc-1700-4fa0-9244-d4a4a97a852c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:03.599473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e09fc-1700-4fa0-9244-d4a4a97a852c None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:03.601264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 113/235] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:03.606938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3201e1bf-ba4b-437b-add1-70900bc6909d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:03.607338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3201e1bf-ba4b-437b-add1-70900bc6909d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:03.607526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3201e1bf-ba4b-437b-add1-70900bc6909d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:03.632051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3201e1bf-ba4b-437b-add1-70900bc6909d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:03.633281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3201e1bf-ba4b-437b-add1-70900bc6909d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:03.707864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 123/236] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:03.843058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e65e0-f06f-411d-9129-a8859f96f73e None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:03.843724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e65e0-f06f-411d-9129-a8859f96f73e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:03.844192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4e65e0-f06f-411d-9129-a8859f96f73e None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:03.845998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 114/237] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:03.851570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d370a5e-d9a8-4275-8980-a61c5c1d7970 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:03.851818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d370a5e-d9a8-4275-8980-a61c5c1d7970 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:03.852096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d370a5e-d9a8-4275-8980-a61c5c1d7970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:03.880750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-0d370a5e-d9a8-4275-8980-a61c5c1d7970 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:03.881952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0d370a5e-d9a8-4275-8980-a61c5c1d7970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:03.965448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:03.976532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 124/238] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:04.137669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4e65e0-f06f-411d-9129-a8859f96f73e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:04.261665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4e65e0-f06f-411d-9129-a8859f96f73e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:04.290626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d370a5e-d9a8-4275-8980-a61c5c1d7970 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:04.320602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d6df23-2de7-4e4d-a74b-722fe75ad6ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:04.321245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d6df23-2de7-4e4d-a74b-722fe75ad6ee None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:04.321520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d6df23-2de7-4e4d-a74b-722fe75ad6ee None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:04.321791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d6df23-2de7-4e4d-a74b-722fe75ad6ee None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:04.346313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d6df23-2de7-4e4d-a74b-722fe75ad6ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-50d6df23-2de7-4e4d-a74b-722fe75ad6ee None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:04.356328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:04.357219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:04.358001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:04.360070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 115/239] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:04.367551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d6df23-2de7-4e4d-a74b-722fe75ad6ee None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:04.421002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:04.482321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d6df23-2de7-4e4d-a74b-722fe75ad6ee None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:04.518269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:04.519142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:04.520190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:04.520788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:04.523716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:04.523716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:04.523716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:04.532414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 116/240] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:04.539451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:04.541460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d370a5e-d9a8-4275-8980-a61c5c1d7970 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:04.610957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02600b0d-8b9a-48df-945b-60ffde2c61de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:04.612436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02600b0d-8b9a-48df-945b-60ffde2c61de None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:04.612857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02600b0d-8b9a-48df-945b-60ffde2c61de None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:04.613798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02600b0d-8b9a-48df-945b-60ffde2c61de None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:04.615049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02600b0d-8b9a-48df-945b-60ffde2c61de None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:04.618191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02600b0d-8b9a-48df-945b-60ffde2c61de None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:04.622071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 125/241] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:04.626537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02600b0d-8b9a-48df-945b-60ffde2c61de None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:04.688640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d613b7-3315-44c1-a613-a8af16051c32 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:04.691179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:04.734849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffe637d-8053-4a9f-8083-d4d61ea677d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:04.735494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffe637d-8053-4a9f-8083-d4d61ea677d7 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:04.735798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffe637d-8053-4a9f-8083-d4d61ea677d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:04.736075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffe637d-8053-4a9f-8083-d4d61ea677d7 None admin] PATH_INFO: `/v3/users/50b6e1fbef254da6b3853dd07cb2d73c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:04.741037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:04.761115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffe637d-8053-4a9f-8083-d4d61ea677d7 None admin] RBAC: Authorizing `identity:get_user(user_id=50b6e1fbef254da6b3853dd07cb2d73c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:04.762496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffe637d-8053-4a9f-8083-d4d61ea677d7 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:04.764201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 117/242] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:04 2022] GET /identity/v3/users/50b6e1fbef254da6b3853dd07cb2d73c => generated 297 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:04.770296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffe637d-8053-4a9f-8083-d4d61ea677d7 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:04.818530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02600b0d-8b9a-48df-945b-60ffde2c61de None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:04.875289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffe637d-8053-4a9f-8083-d4d61ea677d7 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:04.885472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900270ac-e6fe-4108-96a0-7335aad094fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:04.886107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900270ac-e6fe-4108-96a0-7335aad094fe None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:04.886391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900270ac-e6fe-4108-96a0-7335aad094fe None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:04.886672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900270ac-e6fe-4108-96a0-7335aad094fe None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:04.887340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900270ac-e6fe-4108-96a0-7335aad094fe None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:04.888912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900270ac-e6fe-4108-96a0-7335aad094fe None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:04.916721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3707f611-3c7b-4cdc-b5f6-9113bddd8242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:04.917350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3707f611-3c7b-4cdc-b5f6-9113bddd8242 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:04.917633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3707f611-3c7b-4cdc-b5f6-9113bddd8242 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:04.917972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3707f611-3c7b-4cdc-b5f6-9113bddd8242 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:04.936140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 126/243] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:56:04 2022] POST /identity/v3/users => generated 293 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:56:04.961433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:05.018869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3707f611-3c7b-4cdc-b5f6-9113bddd8242 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:05.020565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3707f611-3c7b-4cdc-b5f6-9113bddd8242 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:05.136360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3ef9cf-0be7-4cd0-ad9e-9cf28588fbf4 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.136935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3ef9cf-0be7-4cd0-ad9e-9cf28588fbf4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.137244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3ef9cf-0be7-4cd0-ad9e-9cf28588fbf4 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.138823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 127/244] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:05.139907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 118/245] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:05.145612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa4a58d-60d6-46ae-b8ab-faf1e06a9510 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.145759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3707f611-3c7b-4cdc-b5f6-9113bddd8242 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:05.145945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa4a58d-60d6-46ae-b8ab-faf1e06a9510 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.146176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa4a58d-60d6-46ae-b8ab-faf1e06a9510 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.169088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5aa4a58d-60d6-46ae-b8ab-faf1e06a9510 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:05.170315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5aa4a58d-60d6-46ae-b8ab-faf1e06a9510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:05.228179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:05.260204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 128/246] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:05.277082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3707f611-3c7b-4cdc-b5f6-9113bddd8242 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:05.317849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd18b7-2948-49ff-b3fc-0d882987a148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:05.318725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd18b7-2948-49ff-b3fc-0d882987a148 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.319199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd18b7-2948-49ff-b3fc-0d882987a148 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.319731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd18b7-2948-49ff-b3fc-0d882987a148 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.321073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbd18b7-2948-49ff-b3fc-0d882987a148 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:05.322728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbd18b7-2948-49ff-b3fc-0d882987a148 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:05.324961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 119/247] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:05 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:05.330461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa4a58d-60d6-46ae-b8ab-faf1e06a9510 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:05.387217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:05.447953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa4a58d-60d6-46ae-b8ab-faf1e06a9510 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:05.498369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:05.498924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.499206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.499481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.500269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] RBAC: Adding query filter params (role.id=9950301e6a2d4ba594bc506ec017aad0, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7, user.id=50b6e1fbef254da6b3853dd07cb2d73c) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:05.500578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:05.502019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:05.508458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 129/248] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:05 2022] GET /identity/v3/role_assignments?user.id=50b6e1fbef254da6b3853dd07cb2d73c&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=9950301e6a2d4ba594bc506ec017aad0 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:05.571893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bad82b-802b-4a2a-80ce-3b9787d5e2a2 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.572348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bad82b-802b-4a2a-80ce-3b9787d5e2a2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.572722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bad82b-802b-4a2a-80ce-3b9787d5e2a2 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.572831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:05.574480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 120/249] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:05.580536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75964c2a-6160-4c55-8a45-1086938fb66c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.580923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75964c2a-6160-4c55-8a45-1086938fb66c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.581203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75964c2a-6160-4c55-8a45-1086938fb66c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.605065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-75964c2a-6160-4c55-8a45-1086938fb66c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:05.606385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-75964c2a-6160-4c55-8a45-1086938fb66c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:05.692730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84eeb0c-701c-4866-be25-6b81b8c33b46 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:05.700838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 121/250] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:05.702765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9911d586-80ea-4d01-bb0d-687e7daf86e5 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.703221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9911d586-80ea-4d01-bb0d-687e7daf86e5 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.703663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9911d586-80ea-4d01-bb0d-687e7daf86e5 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.709357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:05.709799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 122/251] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:05 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:05.718169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe523bf4-c62b-4f5c-9902-c01699529f2d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.718605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe523bf4-c62b-4f5c-9902-c01699529f2d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.718950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe523bf4-c62b-4f5c-9902-c01699529f2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.740909 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-fe523bf4-c62b-4f5c-9902-c01699529f2d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:05.742178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fe523bf4-c62b-4f5c-9902-c01699529f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:05.743886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bcb126-b7af-41df-ad22-69ca648a872e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:05.744754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcb126-b7af-41df-ad22-69ca648a872e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.745208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcb126-b7af-41df-ad22-69ca648a872e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.745652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcb126-b7af-41df-ad22-69ca648a872e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.760693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bcb126-b7af-41df-ad22-69ca648a872e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-18bcb126-b7af-41df-ad22-69ca648a872e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:05.767183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:05.768166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:05.768969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:05.773579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 130/252] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:05.774751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bcb126-b7af-41df-ad22-69ca648a872e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:05.813718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:05.833709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 123/253] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:05.903193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bcb126-b7af-41df-ad22-69ca648a872e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:05.945139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:05.945759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:05.946040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:05.946372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:05.947053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:05.947357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:05.948788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:05.963948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 131/254] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:05.969654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe523bf4-c62b-4f5c-9902-c01699529f2d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:06.057627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:06.122461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe523bf4-c62b-4f5c-9902-c01699529f2d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:06.123110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:06.207483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd31590-e8d0-467f-a94b-62057f03813b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:06.208156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd31590-e8d0-467f-a94b-62057f03813b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:06.208539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd31590-e8d0-467f-a94b-62057f03813b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:06.208870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd31590-e8d0-467f-a94b-62057f03813b None admin] PATH_INFO: `/v3/users/628f692df03d49a8a23f574d27e02b09` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:06.212450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd31590-e8d0-467f-a94b-62057f03813b None admin] RBAC: Authorizing `identity:get_user(user_id=628f692df03d49a8a23f574d27e02b09)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:06.213912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd31590-e8d0-467f-a94b-62057f03813b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:06.215879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 124/255] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:05 2022] GET /identity/v3/users/628f692df03d49a8a23f574d27e02b09 => generated 297 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:06.221220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd31590-e8d0-467f-a94b-62057f03813b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:06.246423 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c951120-0f68-42db-9ad1-fa6b3c1b8ebf None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:06.320262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0a6008-23df-49dd-a757-deb8c61c816c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:06.320906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0a6008-23df-49dd-a757-deb8c61c816c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:06.321183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0a6008-23df-49dd-a757-deb8c61c816c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:06.321454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0a6008-23df-49dd-a757-deb8c61c816c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:06.341433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0a6008-23df-49dd-a757-deb8c61c816c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-fc0a6008-23df-49dd-a757-deb8c61c816c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:06.348446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:06.349403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:06.350199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 132/256] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:06.351484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0a6008-23df-49dd-a757-deb8c61c816c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:06.362472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:06.454351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd31590-e8d0-467f-a94b-62057f03813b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:06.508060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:06.516312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0a6008-23df-49dd-a757-deb8c61c816c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:06.517993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0307550-ba5f-450f-a0e5-a992b68f51a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:06.518849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0307550-ba5f-450f-a0e5-a992b68f51a9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:06.519370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0307550-ba5f-450f-a0e5-a992b68f51a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:06.519912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0307550-ba5f-450f-a0e5-a992b68f51a9 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:06.521378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0307550-ba5f-450f-a0e5-a992b68f51a9 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:06.523019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0307550-ba5f-450f-a0e5-a992b68f51a9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:06.525231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 125/257] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:06 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:06.526187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0307550-ba5f-450f-a0e5-a992b68f51a9 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:06.608614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb832db-77ca-47cc-b46c-e069ea1d590d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:06.609697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb832db-77ca-47cc-b46c-e069ea1d590d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:06.609697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb832db-77ca-47cc-b46c-e069ea1d590d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:06.609697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb832db-77ca-47cc-b46c-e069ea1d590d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:06.637825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb832db-77ca-47cc-b46c-e069ea1d590d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3cb832db-77ca-47cc-b46c-e069ea1d590d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:06.643545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:06.644577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:06.645465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:06.648080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 133/258] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:56:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:06.649850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb832db-77ca-47cc-b46c-e069ea1d590d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:06.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0307550-ba5f-450f-a0e5-a992b68f51a9 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:06.717356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:06.760951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:06.762597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:06.763605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:06.764264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:06.766100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:06.767388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb832db-77ca-47cc-b46c-e069ea1d590d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:06.768305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:06.770699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:06.793735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 126/259] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:06.795057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:06.844326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c0c4e-8c25-444f-8c2c-f78a99299a9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:06.846586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c0c4e-8c25-444f-8c2c-f78a99299a9c None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:06.847068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c0c4e-8c25-444f-8c2c-f78a99299a9c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:06.848948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c0c4e-8c25-444f-8c2c-f78a99299a9c None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/users/628f692df03d49a8a23f574d27e02b09/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:06.853538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c0c4e-8c25-444f-8c2c-f78a99299a9c None admin] RBAC: Authorizing `identity:create_grant(project_id=6fa314f8793248fb8bbde6daa676e4e7, user_id=628f692df03d49a8a23f574d27e02b09, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:06.856508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c0c4e-8c25-444f-8c2c-f78a99299a9c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:06.887181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 134/260] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:06 2022] PUT /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/users/628f692df03d49a8a23f574d27e02b09/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:06.888457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c0c4e-8c25-444f-8c2c-f78a99299a9c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:06.916073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:06.969166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4018db-53cb-4c12-8bd9-b50ae0472a2d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:07.008065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:07.009060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:07.009539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:07.010029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:07.010931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:07.011469 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:07.013161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:07.023567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 127/261] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:07.029101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:07.084651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c0c4e-8c25-444f-8c2c-f78a99299a9c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:07.086060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:07.142917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b6625c-76e1-476b-b035-7e740f162a20 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:07.161961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b50a55-b639-4165-b28e-7e4524bb7190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:07.162783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b50a55-b639-4165-b28e-7e4524bb7190 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:07.163158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b50a55-b639-4165-b28e-7e4524bb7190 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:07.163549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b50a55-b639-4165-b28e-7e4524bb7190 None admin] PATH_INFO: `/v3/users/4c6fa3433f69465187738d533e6044a2` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:07.166982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b50a55-b639-4165-b28e-7e4524bb7190 None admin] RBAC: Authorizing `identity:get_user(user_id=4c6fa3433f69465187738d533e6044a2)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:07.171709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b50a55-b639-4165-b28e-7e4524bb7190 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:07.171709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 135/262] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:06 2022] GET /identity/v3/users/4c6fa3433f69465187738d533e6044a2 => generated 283 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:07.176780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b50a55-b639-4165-b28e-7e4524bb7190 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:07.198597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68faa12-04c4-4d35-aa32-9cf455923d19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:07.199575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68faa12-04c4-4d35-aa32-9cf455923d19 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:07.200017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68faa12-04c4-4d35-aa32-9cf455923d19 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:07.200482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68faa12-04c4-4d35-aa32-9cf455923d19 None admin] PATH_INFO: `/v3/users/4c6fa3433f69465187738d533e6044a2` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:07.202283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68faa12-04c4-4d35-aa32-9cf455923d19 None admin] RBAC: Authorizing `identity:get_user(user_id=4c6fa3433f69465187738d533e6044a2)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:07.204022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68faa12-04c4-4d35-aa32-9cf455923d19 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:07.206209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 128/263] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:07 2022] GET /identity/v3/users/4c6fa3433f69465187738d533e6044a2 => generated 283 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:07.212677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68faa12-04c4-4d35-aa32-9cf455923d19 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:07.336036 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68faa12-04c4-4d35-aa32-9cf455923d19 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:07.346879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:07.369431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:07.376866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b50a55-b639-4165-b28e-7e4524bb7190 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:07.388420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b52e69f-7609-409a-a668-8f2b62f19848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:07.389054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52e69f-7609-409a-a668-8f2b62f19848 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:07.389332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52e69f-7609-409a-a668-8f2b62f19848 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:07.389606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52e69f-7609-409a-a668-8f2b62f19848 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:07.444485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc46074-68f2-4be0-b3b6-2d6faea647d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:07.445188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc46074-68f2-4be0-b3b6-2d6faea647d5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:07.445563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc46074-68f2-4be0-b3b6-2d6faea647d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:07.445928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc46074-68f2-4be0-b3b6-2d6faea647d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:07.538065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b52e69f-7609-409a-a668-8f2b62f19848 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:07.539528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b52e69f-7609-409a-a668-8f2b62f19848 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:07.592825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc46074-68f2-4be0-b3b6-2d6faea647d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:07.594318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc46074-68f2-4be0-b3b6-2d6faea647d5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:07.705442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 129/264] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:07.716375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b52e69f-7609-409a-a668-8f2b62f19848 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:07.748061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:07.829990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:07.872065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 136/265] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 696 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:07.878821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc46074-68f2-4be0-b3b6-2d6faea647d5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:07.911315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b52e69f-7609-409a-a668-8f2b62f19848 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:07.968483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f7ef6-7de4-4c72-a20b-a5a5e1d10590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:07.969503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f7ef6-7de4-4c72-a20b-a5a5e1d10590 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:07.969980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f7ef6-7de4-4c72-a20b-a5a5e1d10590 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:07.970480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f7ef6-7de4-4c72-a20b-a5a5e1d10590 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:07.972982 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08f7ef6-7de4-4c72-a20b-a5a5e1d10590 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:07.974686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08f7ef6-7de4-4c72-a20b-a5a5e1d10590 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:07.980914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 130/266] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:07 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:07.990132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f7ef6-7de4-4c72-a20b-a5a5e1d10590 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:08.069625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc46074-68f2-4be0-b3b6-2d6faea647d5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:08.139065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34992311-f748-48a7-8742-10700e1de62e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:08.139781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34992311-f748-48a7-8742-10700e1de62e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.140075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34992311-f748-48a7-8742-10700e1de62e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.140369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34992311-f748-48a7-8742-10700e1de62e None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.142510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34992311-f748-48a7-8742-10700e1de62e None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:08.144084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34992311-f748-48a7-8742-10700e1de62e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:08.146161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 137/267] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:07 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:08.151842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34992311-f748-48a7-8742-10700e1de62e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:08.153050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:08.176599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f7ef6-7de4-4c72-a20b-a5a5e1d10590 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:08.241883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1522dc73-d488-4f55-806c-f2a94c7b4281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:08.242425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1522dc73-d488-4f55-806c-f2a94c7b4281 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.242702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1522dc73-d488-4f55-806c-f2a94c7b4281 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.242970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1522dc73-d488-4f55-806c-f2a94c7b4281 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.243712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1522dc73-d488-4f55-806c-f2a94c7b4281 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7, user.id=4c6fa3433f69465187738d533e6044a2) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:08.244040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1522dc73-d488-4f55-806c-f2a94c7b4281 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:08.245472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1522dc73-d488-4f55-806c-f2a94c7b4281 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:08.264054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:08.264406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 131/268] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:07 2022] GET /identity/v3/role_assignments?user.id=4c6fa3433f69465187738d533e6044a2&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 608 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:08.265641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b636f22-249d-41a4-9929-b74f19c5935a None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.267063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b636f22-249d-41a4-9929-b74f19c5935a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.267348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b636f22-249d-41a4-9929-b74f19c5935a None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.271408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 132/269] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:08.280774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c55ca07-5a36-4d11-915e-3a8fe1bbe490 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.281055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c55ca07-5a36-4d11-915e-3a8fe1bbe490 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.281330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c55ca07-5a36-4d11-915e-3a8fe1bbe490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.325449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-5c55ca07-5a36-4d11-915e-3a8fe1bbe490 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:08.326663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5c55ca07-5a36-4d11-915e-3a8fe1bbe490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:08.374164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34992311-f748-48a7-8742-10700e1de62e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:08.463168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca12f43-38a1-49a2-a3ab-1bfb0a185c3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:08.463810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca12f43-38a1-49a2-a3ab-1bfb0a185c3f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.464092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca12f43-38a1-49a2-a3ab-1bfb0a185c3f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.464359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca12f43-38a1-49a2-a3ab-1bfb0a185c3f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.465098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca12f43-38a1-49a2-a3ab-1bfb0a185c3f None admin] RBAC: Adding query filter params (role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7, user.id=4c6fa3433f69465187738d533e6044a2) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:08.465404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca12f43-38a1-49a2-a3ab-1bfb0a185c3f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:08.466971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca12f43-38a1-49a2-a3ab-1bfb0a185c3f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:08.474075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 138/270] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:08 2022] GET /identity/v3/role_assignments?user.id=4c6fa3433f69465187738d533e6044a2&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 608 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:08.493641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 133/271] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:08.827140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961b7855-c292-4366-8915-67ea3edd4411 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.827341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961b7855-c292-4366-8915-67ea3edd4411 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.827865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961b7855-c292-4366-8915-67ea3edd4411 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.829745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 139/272] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:08.835568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821f1c85-ea2a-4f13-9097-5d16d98dc48f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.836455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821f1c85-ea2a-4f13-9097-5d16d98dc48f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.836934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821f1c85-ea2a-4f13-9097-5d16d98dc48f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.863121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-821f1c85-ea2a-4f13-9097-5d16d98dc48f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:08.864810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-821f1c85-ea2a-4f13-9097-5d16d98dc48f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:08.882940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3280ac-643b-4e1f-b57d-4fd67e4636e7 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.883432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3280ac-643b-4e1f-b57d-4fd67e4636e7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.883536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3280ac-643b-4e1f-b57d-4fd67e4636e7 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.885006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 140/273] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:08.891276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49aa6b1-9559-4662-8a37-1e3345568c98 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:08.891562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49aa6b1-9559-4662-8a37-1e3345568c98 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:08.891878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49aa6b1-9559-4662-8a37-1e3345568c98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:08.917319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-f49aa6b1-9559-4662-8a37-1e3345568c98 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:08.918637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f49aa6b1-9559-4662-8a37-1e3345568c98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:08.970725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 134/274] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:08.972115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821f1c85-ea2a-4f13-9097-5d16d98dc48f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:08.987354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:08.998428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 141/275] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:09.084553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821f1c85-ea2a-4f13-9097-5d16d98dc48f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.120091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a868e73-fa8a-4136-9f26-68a9a5d34218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.120722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a868e73-fa8a-4136-9f26-68a9a5d34218 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.121005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a868e73-fa8a-4136-9f26-68a9a5d34218 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.121279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a868e73-fa8a-4136-9f26-68a9a5d34218 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.131961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a868e73-fa8a-4136-9f26-68a9a5d34218 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-5a868e73-fa8a-4136-9f26-68a9a5d34218 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:09.136745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:09.137648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:09.138450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 135/276] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:08 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:09.146663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49aa6b1-9559-4662-8a37-1e3345568c98 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.194822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:09.251896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49aa6b1-9559-4662-8a37-1e3345568c98 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.261042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a868e73-fa8a-4136-9f26-68a9a5d34218 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.289368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.289996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.290274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.290546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.291216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:09.291616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:09.292932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:09.303883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 142/277] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:09 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:09.308639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.342182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:09.374681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a868e73-fa8a-4136-9f26-68a9a5d34218 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.414191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e79d71-7a06-4a5a-b311-fe5316c33767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.414787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e79d71-7a06-4a5a-b311-fe5316c33767 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.415116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e79d71-7a06-4a5a-b311-fe5316c33767 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.415402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e79d71-7a06-4a5a-b311-fe5316c33767 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.416609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e79d71-7a06-4a5a-b311-fe5316c33767 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:09.417970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e79d71-7a06-4a5a-b311-fe5316c33767 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:09.419365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a6b8e-baee-4317-b46b-2f1d4aa8e205 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.424334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 136/278] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:09.425081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e79d71-7a06-4a5a-b311-fe5316c33767 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.427772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:09.456884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3458cc7-f5a1-44fd-8a42-288c5db17f02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.457689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3458cc7-f5a1-44fd-8a42-288c5db17f02 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.457989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3458cc7-f5a1-44fd-8a42-288c5db17f02 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.458265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3458cc7-f5a1-44fd-8a42-288c5db17f02 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.468921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3458cc7-f5a1-44fd-8a42-288c5db17f02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a3458cc7-f5a1-44fd-8a42-288c5db17f02 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:09.474997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:09.475912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:09.476727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 143/279] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:09.477373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3458cc7-f5a1-44fd-8a42-288c5db17f02 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.538509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e79d71-7a06-4a5a-b311-fe5316c33767 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.575626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31893230-1c68-480c-99d1-529a7dc82369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.576287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31893230-1c68-480c-99d1-529a7dc82369 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.576567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31893230-1c68-480c-99d1-529a7dc82369 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.576841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31893230-1c68-480c-99d1-529a7dc82369 None admin] PATH_INFO: `/v3/users/50b6e1fbef254da6b3853dd07cb2d73c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.578703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31893230-1c68-480c-99d1-529a7dc82369 None admin] RBAC: Authorizing `identity:get_user(user_id=50b6e1fbef254da6b3853dd07cb2d73c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:09.580512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31893230-1c68-480c-99d1-529a7dc82369 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:09.581201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3458cc7-f5a1-44fd-8a42-288c5db17f02 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.587272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 137/280] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:09 2022] GET /identity/v3/users/50b6e1fbef254da6b3853dd07cb2d73c => generated 297 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:09.587272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31893230-1c68-480c-99d1-529a7dc82369 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.596044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:09.628548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b2b59e-39a1-4c91-82bf-56a9dd0257d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.629992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b2b59e-39a1-4c91-82bf-56a9dd0257d6 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.630281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b2b59e-39a1-4c91-82bf-56a9dd0257d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.630555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b2b59e-39a1-4c91-82bf-56a9dd0257d6 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.631232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b2b59e-39a1-4c91-82bf-56a9dd0257d6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:09.632698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b2b59e-39a1-4c91-82bf-56a9dd0257d6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:09.671719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 144/281] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:56:09 2022] POST /identity/v3/users => generated 293 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:56:09.671719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b2b59e-39a1-4c91-82bf-56a9dd0257d6 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.702332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:09.716380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31893230-1c68-480c-99d1-529a7dc82369 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.756354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3834f4-ef13-4a10-ac1b-94e683a396f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.756977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3834f4-ef13-4a10-ac1b-94e683a396f3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.757255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3834f4-ef13-4a10-ac1b-94e683a396f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.757527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3834f4-ef13-4a10-ac1b-94e683a396f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.758212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3834f4-ef13-4a10-ac1b-94e683a396f3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:09.758518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3834f4-ef13-4a10-ac1b-94e683a396f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:09.759968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3834f4-ef13-4a10-ac1b-94e683a396f3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:09.775134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 138/282] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:09.775134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9abd25-c293-4138-bba2-b7fe545ba225 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.775134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9abd25-c293-4138-bba2-b7fe545ba225 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.775134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9abd25-c293-4138-bba2-b7fe545ba225 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.775134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 139/283] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:09.776827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9abd25-c293-4138-bba2-b7fe545ba225 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.802758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b2b59e-39a1-4c91-82bf-56a9dd0257d6 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.845016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56d4403-a4c3-476b-86db-36859b97115c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.845016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d4403-a4c3-476b-86db-36859b97115c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.845016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d4403-a4c3-476b-86db-36859b97115c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.845016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56d4403-a4c3-476b-86db-36859b97115c None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.845413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56d4403-a4c3-476b-86db-36859b97115c None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:09.845734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56d4403-a4c3-476b-86db-36859b97115c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:09.847670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 145/284] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:09 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:09.853075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43613dbf-8fe6-4f2a-baa5-6024d08474ac None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.853075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43613dbf-8fe6-4f2a-baa5-6024d08474ac None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.853075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43613dbf-8fe6-4f2a-baa5-6024d08474ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.853382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:09.869153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-43613dbf-8fe6-4f2a-baa5-6024d08474ac None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:09.870349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-43613dbf-8fe6-4f2a-baa5-6024d08474ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:09.899461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9abd25-c293-4138-bba2-b7fe545ba225 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:09.942555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511b4948-45cb-4fde-aff2-26a4bda6e480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:09.943188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511b4948-45cb-4fde-aff2-26a4bda6e480 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:09.943468 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511b4948-45cb-4fde-aff2-26a4bda6e480 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:09.943807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511b4948-45cb-4fde-aff2-26a4bda6e480 None admin] PATH_INFO: `/v3/users/bb0c79040c314d73be1f0167f3136ef4` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:09.955520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 146/285] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:09.956462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43613dbf-8fe6-4f2a-baa5-6024d08474ac None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.961422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511b4948-45cb-4fde-aff2-26a4bda6e480 None admin] RBAC: Authorizing `identity:get_user(user_id=bb0c79040c314d73be1f0167f3136ef4)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:09.962874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511b4948-45cb-4fde-aff2-26a4bda6e480 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:09.964868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 140/286] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:09 2022] GET /identity/v3/users/bb0c79040c314d73be1f0167f3136ef4 => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:09.971729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511b4948-45cb-4fde-aff2-26a4bda6e480 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:09.989434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:10.081897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43613dbf-8fe6-4f2a-baa5-6024d08474ac None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:10.100168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511b4948-45cb-4fde-aff2-26a4bda6e480 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:10.120778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efc446-7128-40e7-984e-2b1f5e0bcc6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:10.121540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efc446-7128-40e7-984e-2b1f5e0bcc6b None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:10.121820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efc446-7128-40e7-984e-2b1f5e0bcc6b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:10.122090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efc446-7128-40e7-984e-2b1f5e0bcc6b None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/users/50b6e1fbef254da6b3853dd07cb2d73c/roles/9950301e6a2d4ba594bc506ec017aad0` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:10.124180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64efc446-7128-40e7-984e-2b1f5e0bcc6b None admin] RBAC: Authorizing `identity:create_grant(project_id=6fa314f8793248fb8bbde6daa676e4e7, user_id=50b6e1fbef254da6b3853dd07cb2d73c, role_id=9950301e6a2d4ba594bc506ec017aad0)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:10.125631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64efc446-7128-40e7-984e-2b1f5e0bcc6b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:10.130700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:10.143617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:10.144591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:10.145093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:10.145684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 147/287] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:09 2022] PUT /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/users/50b6e1fbef254da6b3853dd07cb2d73c/roles/9950301e6a2d4ba594bc506ec017aad0 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:10.146171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:10.147202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] RBAC: Adding query filter params (role.id=b0e7999c9d0d472195e1c5dc6f4d4978, user.id=bb0c79040c314d73be1f0167f3136ef4, scope.system=all) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:10.148433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:10.150187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:10.156787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 141/288] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Sep 15 13:56:09 2022] GET /identity/v3/role_assignments?user.id=bb0c79040c314d73be1f0167f3136ef4&scope.system=all&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:10.262974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efc446-7128-40e7-984e-2b1f5e0bcc6b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:10.390323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:10.414189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efc446-7128-40e7-984e-2b1f5e0bcc6b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:10.453943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480f740f-5112-49f2-9854-175cba8a45f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:10.454600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480f740f-5112-49f2-9854-175cba8a45f7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:10.454878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480f740f-5112-49f2-9854-175cba8a45f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:10.455154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480f740f-5112-49f2-9854-175cba8a45f7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:10.467925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480f740f-5112-49f2-9854-175cba8a45f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-480f740f-5112-49f2-9854-175cba8a45f7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:10.473441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:10.474326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:10.475158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 148/289] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:10.482688 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:10.615981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa46f05-67d6-48ed-8f4f-bd672855a7f5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:10.668075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:10.669030 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:10.669521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:10.669994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:10.670885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:10.671408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:10.673009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:10.685291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 142/290] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:10.692461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480f740f-5112-49f2-9854-175cba8a45f7 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:10.843570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480f740f-5112-49f2-9854-175cba8a45f7 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:10.890617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcfe711-bb21-4993-b324-2418d9fea9aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:10.891266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcfe711-bb21-4993-b324-2418d9fea9aa None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:10.891548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcfe711-bb21-4993-b324-2418d9fea9aa None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:10.891845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcfe711-bb21-4993-b324-2418d9fea9aa None admin] PATH_INFO: `/v3/users/628f692df03d49a8a23f574d27e02b09` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:10.893103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcfe711-bb21-4993-b324-2418d9fea9aa None admin] RBAC: Authorizing `identity:get_user(user_id=628f692df03d49a8a23f574d27e02b09)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:10.894813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcfe711-bb21-4993-b324-2418d9fea9aa None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:10.896757 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 149/291] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:10 2022] GET /identity/v3/users/628f692df03d49a8a23f574d27e02b09 => generated 297 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:10.902078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:10.987748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:11.058066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016a49cc-7af0-4eb6-8fb0-db6b2bcf043d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:11.106726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a4a288-4411-4f8f-a079-4c3afa2ee269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:11.107701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a4a288-4411-4f8f-a079-4c3afa2ee269 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:11.108207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a4a288-4411-4f8f-a079-4c3afa2ee269 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:11.108682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a4a288-4411-4f8f-a079-4c3afa2ee269 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:11.197602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a4a288-4411-4f8f-a079-4c3afa2ee269 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:11.199431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a4a288-4411-4f8f-a079-4c3afa2ee269 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:11.271808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:11.322048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 143/292] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:11.327716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcfe711-bb21-4993-b324-2418d9fea9aa None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:11.346158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:11.479474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcfe711-bb21-4993-b324-2418d9fea9aa None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:11.523801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb386b6c-e5dc-4115-9fae-5d7518e1f9ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:11.524365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb386b6c-e5dc-4115-9fae-5d7518e1f9ef None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:11.524647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb386b6c-e5dc-4115-9fae-5d7518e1f9ef None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:11.524918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb386b6c-e5dc-4115-9fae-5d7518e1f9ef None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:11.526028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb386b6c-e5dc-4115-9fae-5d7518e1f9ef None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:11.527462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb386b6c-e5dc-4115-9fae-5d7518e1f9ef None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:11.529412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 150/293] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:11 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:11.535367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a4a288-4411-4f8f-a079-4c3afa2ee269 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:11.665526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a4a288-4411-4f8f-a079-4c3afa2ee269 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:11.712791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0006a9-9f00-4d76-a2d2-e3c2b9e08e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:11.713451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0006a9-9f00-4d76-a2d2-e3c2b9e08e89 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:11.713736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0006a9-9f00-4d76-a2d2-e3c2b9e08e89 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:11.714065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0006a9-9f00-4d76-a2d2-e3c2b9e08e89 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:11.714798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0006a9-9f00-4d76-a2d2-e3c2b9e08e89 None admin] RBAC: Adding query filter params (role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7, user.id=628f692df03d49a8a23f574d27e02b09) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:11.715108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0006a9-9f00-4d76-a2d2-e3c2b9e08e89 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:11.716552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0006a9-9f00-4d76-a2d2-e3c2b9e08e89 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:11.727576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 144/294] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:11 2022] GET /identity/v3/role_assignments?user.id=628f692df03d49a8a23f574d27e02b09&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:12.348977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51e69a9-659e-462a-97fc-86b4f8563e7a None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:12.349962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51e69a9-659e-462a-97fc-86b4f8563e7a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:12.350430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51e69a9-659e-462a-97fc-86b4f8563e7a None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:12.352354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 151/295] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:12.357479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688e7a0-6144-4ea6-b678-14afda728c5d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:12.358290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688e7a0-6144-4ea6-b678-14afda728c5d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:12.358775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688e7a0-6144-4ea6-b678-14afda728c5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:12.381483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9688e7a0-6144-4ea6-b678-14afda728c5d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:12.382986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9688e7a0-6144-4ea6-b678-14afda728c5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:12.397389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:12.461204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 145/296] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:12.729844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51e69a9-659e-462a-97fc-86b4f8563e7a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:12.771759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2a4294-749b-4391-9630-1561a773850a None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:12.772389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2a4294-749b-4391-9630-1561a773850a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:12.772853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2a4294-749b-4391-9630-1561a773850a None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:12.774382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 146/297] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:12.782860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bde9b9-cf13-4ede-9f67-ce03fd32fe43 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:12.782860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bde9b9-cf13-4ede-9f67-ce03fd32fe43 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:12.782860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bde9b9-cf13-4ede-9f67-ce03fd32fe43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:12.805382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-20bde9b9-cf13-4ede-9f67-ce03fd32fe43 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:12.806630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-20bde9b9-cf13-4ede-9f67-ce03fd32fe43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:12.819500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:12.847392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51e69a9-659e-462a-97fc-86b4f8563e7a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:12.884314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 147/298] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:12.884494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ff223-ab02-4856-9ef8-0c72af5c8d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:12.885094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ff223-ab02-4856-9ef8-0c72af5c8d65 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:12.885455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ff223-ab02-4856-9ef8-0c72af5c8d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:12.885738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ff223-ab02-4856-9ef8-0c72af5c8d65 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:12.886331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d759b4f-52f0-4881-ac0f-f55d4225942e None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:12.886738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d759b4f-52f0-4881-ac0f-f55d4225942e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:12.887135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d759b4f-52f0-4881-ac0f-f55d4225942e None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:12.888777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 148/299] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:12.897866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346ff223-ab02-4856-9ef8-0c72af5c8d65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-346ff223-ab02-4856-9ef8-0c72af5c8d65 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:12.902733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:12.903612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:12.904471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 152/300] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:12.905120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9812ecd4-c9ea-4503-8058-8446f717be1a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:12.905399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9812ecd4-c9ea-4503-8058-8446f717be1a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:12.907729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9812ecd4-c9ea-4503-8058-8446f717be1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:12.916784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ff223-ab02-4856-9ef8-0c72af5c8d65 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:12.931494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9812ecd4-c9ea-4503-8058-8446f717be1a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:12.932738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9812ecd4-c9ea-4503-8058-8446f717be1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:13.002278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 149/301] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:13.003442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ef4a6-0605-412d-9683-0f0264770ec8 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.003784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ef4a6-0605-412d-9683-0f0264770ec8 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.004064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439ef4a6-0605-412d-9683-0f0264770ec8 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.005499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 150/302] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:13.010748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2076a-1cfb-4ffc-a2ae-864b14b39243 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.011039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2076a-1cfb-4ffc-a2ae-864b14b39243 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.011318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2076a-1cfb-4ffc-a2ae-864b14b39243 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.026893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:13.029402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ff223-ab02-4856-9ef8-0c72af5c8d65 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.034180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-24f2076a-1cfb-4ffc-a2ae-864b14b39243 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:13.035663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-24f2076a-1cfb-4ffc-a2ae-864b14b39243 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:13.067048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.067902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.068324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.068737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.069637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:13.069944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:13.071382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:13.073585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:13.084955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 153/303] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:13.085984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.120056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 151/304] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:13.197485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12877973-fe7f-4848-83fa-b52d367d246d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.199188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f2076a-1cfb-4ffc-a2ae-864b14b39243 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.243645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bf8970-d8b8-4791-b062-fecb0fb406cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.244737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bf8970-d8b8-4791-b062-fecb0fb406cc None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.245018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bf8970-d8b8-4791-b062-fecb0fb406cc None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.245295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bf8970-d8b8-4791-b062-fecb0fb406cc None admin] PATH_INFO: `/v3/users/63a404f4799147c2b07ac694a13bd48b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.263547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bf8970-d8b8-4791-b062-fecb0fb406cc None admin] RBAC: Authorizing `identity:get_user(user_id=63a404f4799147c2b07ac694a13bd48b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:13.265593 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bf8970-d8b8-4791-b062-fecb0fb406cc None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:13.268084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 154/305] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/users/63a404f4799147c2b07ac694a13bd48b => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:13.273631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bf8970-d8b8-4791-b062-fecb0fb406cc None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.312011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f2076a-1cfb-4ffc-a2ae-864b14b39243 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.339821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:13.349462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b5a3ed-f27d-4b19-87c1-8fe1ef0ac6a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.350200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b5a3ed-f27d-4b19-87c1-8fe1ef0ac6a7 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.350376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b5a3ed-f27d-4b19-87c1-8fe1ef0ac6a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.350651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b5a3ed-f27d-4b19-87c1-8fe1ef0ac6a7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.351780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b5a3ed-f27d-4b19-87c1-8fe1ef0ac6a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:13.353219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b5a3ed-f27d-4b19-87c1-8fe1ef0ac6a7 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:13.354952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 152/306] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:13.357415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b5a3ed-f27d-4b19-87c1-8fe1ef0ac6a7 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.362613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:13.388691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bf8970-d8b8-4791-b062-fecb0fb406cc None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.424429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c49ea-5dd1-4e22-8d83-761d7e22deed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.425051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c49ea-5dd1-4e22-8d83-761d7e22deed None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.425377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c49ea-5dd1-4e22-8d83-761d7e22deed None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.425657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c49ea-5dd1-4e22-8d83-761d7e22deed None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.435784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773c49ea-5dd1-4e22-8d83-761d7e22deed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-773c49ea-5dd1-4e22-8d83-761d7e22deed None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:13.440531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:13.442002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:13.442789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 155/307] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:13.443474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c49ea-5dd1-4e22-8d83-761d7e22deed None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.466392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b5a3ed-f27d-4b19-87c1-8fe1ef0ac6a7 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.499526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.500167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.500450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.500724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.501456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] RBAC: Adding query filter params (role.id=9950301e6a2d4ba594bc506ec017aad0, user.id=63a404f4799147c2b07ac694a13bd48b, scope.system=all) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:13.501765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:13.503212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:13.508789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 153/308] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/role_assignments?user.id=63a404f4799147c2b07ac694a13bd48b&scope.system=all&role.id=9950301e6a2d4ba594bc506ec017aad0 => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:13.509562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.559400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c49ea-5dd1-4e22-8d83-761d7e22deed None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.582391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:13.587360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:13.603848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1396c452-912f-4803-b3a0-20395930fda4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.604601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1396c452-912f-4803-b3a0-20395930fda4 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.604885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1396c452-912f-4803-b3a0-20395930fda4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.605280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1396c452-912f-4803-b3a0-20395930fda4 None admin] PATH_INFO: `/v3/groups` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.605936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1396c452-912f-4803-b3a0-20395930fda4 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:13.607435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1396c452-912f-4803-b3a0-20395930fda4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:13.625579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 156/309] 10.222.0.27 () {60 vars in 1134 bytes} [Thu Sep 15 13:56:13 2022] POST /identity/v3/groups => generated 229 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:56:13.626295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f3ce9-74f4-4c85-bc44-e8af7d1474f8 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.626473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1396c452-912f-4803-b3a0-20395930fda4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.674071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d44a5e3-d7f9-479d-b298-481260eaf55e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.675135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d44a5e3-d7f9-479d-b298-481260eaf55e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.676428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d44a5e3-d7f9-479d-b298-481260eaf55e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.677165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d44a5e3-d7f9-479d-b298-481260eaf55e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.692825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d44a5e3-d7f9-479d-b298-481260eaf55e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-4d44a5e3-d7f9-479d-b298-481260eaf55e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:13.699508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:13.700660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:13.701477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:13.703224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 154/310] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:13.710018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d44a5e3-d7f9-479d-b298-481260eaf55e None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.790764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1396c452-912f-4803-b3a0-20395930fda4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.839885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.840859 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.840988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d44a5e3-d7f9-479d-b298-481260eaf55e None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:13.841161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.841409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.842085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:13.842391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:13.844587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:13.853199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 157/311] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:13.860041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.889756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:13.890380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:13.890663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:13.890937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:13.891608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:13.891946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:13.892428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:13.900422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:13.907251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:13.919977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 155/312] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:13.921903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:13.978301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e85afb8-d0ff-4857-a08e-0f02ea1115ca None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:14.016083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd91797-17e7-47f4-876f-eea8320af247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:14.016979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd91797-17e7-47f4-876f-eea8320af247 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:14.017333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd91797-17e7-47f4-876f-eea8320af247 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:14.017699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd91797-17e7-47f4-876f-eea8320af247 None admin] PATH_INFO: `/v3/users/bb0c79040c314d73be1f0167f3136ef4` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:14.018961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd91797-17e7-47f4-876f-eea8320af247 None admin] RBAC: Authorizing `identity:get_user(user_id=bb0c79040c314d73be1f0167f3136ef4)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:14.020693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd91797-17e7-47f4-876f-eea8320af247 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:14.022783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 158/313] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/users/bb0c79040c314d73be1f0167f3136ef4 => generated 293 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:14.028053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd91797-17e7-47f4-876f-eea8320af247 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:14.033053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28375d66-fe8b-4a1d-a405-ebaab3e0848a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:14.072596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99bd80d-d489-4258-b6d1-e6fddc268b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:14.073600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99bd80d-d489-4258-b6d1-e6fddc268b8a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:14.074095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99bd80d-d489-4258-b6d1-e6fddc268b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:14.074567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99bd80d-d489-4258-b6d1-e6fddc268b8a None admin] PATH_INFO: `/v3/users/50b6e1fbef254da6b3853dd07cb2d73c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:14.075956 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99bd80d-d489-4258-b6d1-e6fddc268b8a None admin] RBAC: Authorizing `identity:get_user(user_id=50b6e1fbef254da6b3853dd07cb2d73c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:14.077666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99bd80d-d489-4258-b6d1-e6fddc268b8a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:14.079792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 156/314] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:13 2022] GET /identity/v3/users/50b6e1fbef254da6b3853dd07cb2d73c => generated 297 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:14.085637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99bd80d-d489-4258-b6d1-e6fddc268b8a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:14.145819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd91797-17e7-47f4-876f-eea8320af247 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:14.158473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:14.158627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:14.193328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e0c9be-c57e-442d-9e50-41c085689166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:14.194378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e0c9be-c57e-442d-9e50-41c085689166 None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:14.194877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e0c9be-c57e-442d-9e50-41c085689166 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:14.195796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e0c9be-c57e-442d-9e50-41c085689166 None admin] PATH_INFO: `/v3/system/users/bb0c79040c314d73be1f0167f3136ef4/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:14.197829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e0c9be-c57e-442d-9e50-41c085689166 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=bb0c79040c314d73be1f0167f3136ef4, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:14.199379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e0c9be-c57e-442d-9e50-41c085689166 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:14.216296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 159/315] 10.222.0.27 () {58 vars in 1334 bytes} [Thu Sep 15 13:56:14 2022] PUT /identity/v3/system/users/bb0c79040c314d73be1f0167f3136ef4/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:14.222432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99bd80d-d489-4258-b6d1-e6fddc268b8a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:14.263282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b71dd49-01d6-4b66-a7e3-88ca8a5e0b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:14.264000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71dd49-01d6-4b66-a7e3-88ca8a5e0b41 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:14.264287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71dd49-01d6-4b66-a7e3-88ca8a5e0b41 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:14.264564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71dd49-01d6-4b66-a7e3-88ca8a5e0b41 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:14.355437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b71dd49-01d6-4b66-a7e3-88ca8a5e0b41 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:14.356990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b71dd49-01d6-4b66-a7e3-88ca8a5e0b41 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:14.427157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:14.473343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 157/316] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:56:14.479395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e0c9be-c57e-442d-9e50-41c085689166 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:14.586795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e0c9be-c57e-442d-9e50-41c085689166 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:14.625645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a7fd58-a567-4e0b-a40d-c5bdad78a668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:14.626352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a7fd58-a567-4e0b-a40d-c5bdad78a668 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:14.626640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a7fd58-a567-4e0b-a40d-c5bdad78a668 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:14.626916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a7fd58-a567-4e0b-a40d-c5bdad78a668 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:14.628080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a7fd58-a567-4e0b-a40d-c5bdad78a668 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:14.630126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a7fd58-a567-4e0b-a40d-c5bdad78a668 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:14.636041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 160/317] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:14 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:14.638538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b71dd49-01d6-4b66-a7e3-88ca8a5e0b41 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:14.744986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b71dd49-01d6-4b66-a7e3-88ca8a5e0b41 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:14.782607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690dce25-c05a-49e2-9a84-3bed6f054a7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:14.783505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690dce25-c05a-49e2-9a84-3bed6f054a7b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:14.783980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690dce25-c05a-49e2-9a84-3bed6f054a7b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:14.784453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690dce25-c05a-49e2-9a84-3bed6f054a7b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:14.785675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690dce25-c05a-49e2-9a84-3bed6f054a7b None admin] RBAC: Adding query filter params (role.id=9950301e6a2d4ba594bc506ec017aad0, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7, user.id=50b6e1fbef254da6b3853dd07cb2d73c) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:14.786163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690dce25-c05a-49e2-9a84-3bed6f054a7b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:14.787776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690dce25-c05a-49e2-9a84-3bed6f054a7b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:14.794404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 158/318] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:14 2022] GET /identity/v3/role_assignments?user.id=50b6e1fbef254da6b3853dd07cb2d73c&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=9950301e6a2d4ba594bc506ec017aad0 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:15.718140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e94581-d733-44de-ad1d-64cfd95fc213 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:15.718641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e94581-d733-44de-ad1d-64cfd95fc213 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:15.718715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e94581-d733-44de-ad1d-64cfd95fc213 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:15.720230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 161/319] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:15.725812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01da5df-f48f-410f-aacf-fb339385b33e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:15.726208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01da5df-f48f-410f-aacf-fb339385b33e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:15.726367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01da5df-f48f-410f-aacf-fb339385b33e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:15.744483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b01da5df-f48f-410f-aacf-fb339385b33e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:15.745651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b01da5df-f48f-410f-aacf-fb339385b33e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:15.762354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:15.818469 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 159/320] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:15.930349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f66b17d-e576-4943-9344-9d12a43501d9 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:15.930695 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f66b17d-e576-4943-9344-9d12a43501d9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:15.930912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f66b17d-e576-4943-9344-9d12a43501d9 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:15.932421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 162/321] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:15.937413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca64ef71-8e05-4cf3-a4c6-cf335608d9db None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:15.937707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca64ef71-8e05-4cf3-a4c6-cf335608d9db None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:15.937987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca64ef71-8e05-4cf3-a4c6-cf335608d9db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:15.955924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ca64ef71-8e05-4cf3-a4c6-cf335608d9db None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:15.957633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ca64ef71-8e05-4cf3-a4c6-cf335608d9db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:16.024407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 160/322] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:16.090332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f66b17d-e576-4943-9344-9d12a43501d9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:16.133274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:16.198170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f66b17d-e576-4943-9344-9d12a43501d9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:16.232805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b9acc6-042f-4759-a13a-ac5c51f77586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:16.233437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b9acc6-042f-4759-a13a-ac5c51f77586 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:16.233731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b9acc6-042f-4759-a13a-ac5c51f77586 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:16.233993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b9acc6-042f-4759-a13a-ac5c51f77586 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:16.245564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b9acc6-042f-4759-a13a-ac5c51f77586 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d5b9acc6-042f-4759-a13a-ac5c51f77586 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:16.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:16.251273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:16.252141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 163/323] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:16.259728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca64ef71-8e05-4cf3-a4c6-cf335608d9db None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:16.299837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b9acc6-042f-4759-a13a-ac5c51f77586 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:16.349354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:16.371961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca64ef71-8e05-4cf3-a4c6-cf335608d9db None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:16.411946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:16.412657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:16.412945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:16.413224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:16.413827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:16.413936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:16.414191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:16.415824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:16.418285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b9acc6-042f-4759-a13a-ac5c51f77586 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:16.425000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 161/324] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:16.430621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:16.454912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d7b36a-8e38-4b47-b00d-66176ff07d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:16.455538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d7b36a-8e38-4b47-b00d-66176ff07d36 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:16.455846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d7b36a-8e38-4b47-b00d-66176ff07d36 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:16.456168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d7b36a-8e38-4b47-b00d-66176ff07d36 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:16.457703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d7b36a-8e38-4b47-b00d-66176ff07d36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:16.458937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d7b36a-8e38-4b47-b00d-66176ff07d36 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:16.460709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 164/325] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:16.465804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d7b36a-8e38-4b47-b00d-66176ff07d36 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:16.531507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d936377a-c4fb-4f24-a6b1-38b09be56c76 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:16.563655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d7b36a-8e38-4b47-b00d-66176ff07d36 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:16.565145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f5b61e-ad2e-4dc9-b3fb-d594ed026541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:16.565782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f5b61e-ad2e-4dc9-b3fb-d594ed026541 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:16.566197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f5b61e-ad2e-4dc9-b3fb-d594ed026541 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:16.566472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f5b61e-ad2e-4dc9-b3fb-d594ed026541 None admin] PATH_INFO: `/v3/users/63a404f4799147c2b07ac694a13bd48b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:16.567600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f5b61e-ad2e-4dc9-b3fb-d594ed026541 None admin] RBAC: Authorizing `identity:get_user(user_id=63a404f4799147c2b07ac694a13bd48b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:16.569096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f5b61e-ad2e-4dc9-b3fb-d594ed026541 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:16.570665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 162/326] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:16 2022] GET /identity/v3/users/63a404f4799147c2b07ac694a13bd48b => generated 293 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:16.571812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1938aa-f6b8-4533-8e5b-b62616b3a0dc None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:16.572092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1938aa-f6b8-4533-8e5b-b62616b3a0dc None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:16.572360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1938aa-f6b8-4533-8e5b-b62616b3a0dc None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:16.573723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 163/327] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:16 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:16.576532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1938aa-f6b8-4533-8e5b-b62616b3a0dc None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:16.591880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:16.600753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d25c81-5418-47e0-8434-267f6bee51af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:16.601362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d25c81-5418-47e0-8434-267f6bee51af None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:16.601635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d25c81-5418-47e0-8434-267f6bee51af None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:16.601903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d25c81-5418-47e0-8434-267f6bee51af None admin] PATH_INFO: `/v3/groups` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:16.602527 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d25c81-5418-47e0-8434-267f6bee51af None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:16.604046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d25c81-5418-47e0-8434-267f6bee51af None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:16.619019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 165/328] 10.222.0.27 () {60 vars in 1134 bytes} [Thu Sep 15 13:56:16 2022] POST /identity/v3/groups => generated 226 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:56:16.620435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8ef87-70df-4220-af45-ff2331aaf676 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:16.620718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8ef87-70df-4220-af45-ff2331aaf676 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:16.621003 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8ef87-70df-4220-af45-ff2331aaf676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:16.645639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-67d8ef87-70df-4220-af45-ff2331aaf676 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:16.646911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-67d8ef87-70df-4220-af45-ff2331aaf676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:16.678173 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:16.693920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1938aa-f6b8-4533-8e5b-b62616b3a0dc None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:16.733078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7187fa3-292f-4e49-a72f-45c9839b26c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:16.734533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7187fa3-292f-4e49-a72f-45c9839b26c4 None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:16.735197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7187fa3-292f-4e49-a72f-45c9839b26c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:16.735616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7187fa3-292f-4e49-a72f-45c9839b26c4 None admin] PATH_INFO: `/v3/system/users/63a404f4799147c2b07ac694a13bd48b/roles/9950301e6a2d4ba594bc506ec017aad0` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:16.737449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7187fa3-292f-4e49-a72f-45c9839b26c4 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=63a404f4799147c2b07ac694a13bd48b, role_id=9950301e6a2d4ba594bc506ec017aad0)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:16.740068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7187fa3-292f-4e49-a72f-45c9839b26c4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:16.741114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 166/329] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:16.752291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 164/330] 10.222.0.27 () {58 vars in 1334 bytes} [Thu Sep 15 13:56:16 2022] PUT /identity/v3/system/users/63a404f4799147c2b07ac694a13bd48b/roles/9950301e6a2d4ba594bc506ec017aad0 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:17.020318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d8ef87-70df-4220-af45-ff2331aaf676 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:17.129745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d8ef87-70df-4220-af45-ff2331aaf676 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:17.168062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f20a23-6691-4705-af2a-f42f4a4ddadd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:17.168709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f20a23-6691-4705-af2a-f42f4a4ddadd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:17.168991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f20a23-6691-4705-af2a-f42f4a4ddadd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:17.169266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f20a23-6691-4705-af2a-f42f4a4ddadd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:17.181530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f20a23-6691-4705-af2a-f42f4a4ddadd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-60f20a23-6691-4705-af2a-f42f4a4ddadd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:17.186487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:17.187356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:17.188574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 167/331] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:17.194331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7187fa3-292f-4e49-a72f-45c9839b26c4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:17.285762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:17.317739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7187fa3-292f-4e49-a72f-45c9839b26c4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:17.356145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:17.356758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:17.357037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:17.357310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:17.360277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:17.360591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:17.362038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:17.373928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 165/332] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:17.379806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f20a23-6691-4705-af2a-f42f4a4ddadd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:17.418315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:17.518172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f20a23-6691-4705-af2a-f42f4a4ddadd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:17.557767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dbd63c-a2e9-42c5-ac03-3545f6c8f78e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:17.558326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dbd63c-a2e9-42c5-ac03-3545f6c8f78e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:17.558649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dbd63c-a2e9-42c5-ac03-3545f6c8f78e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:17.558926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dbd63c-a2e9-42c5-ac03-3545f6c8f78e None admin] PATH_INFO: `/v3/users/bb0c79040c314d73be1f0167f3136ef4` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:17.560409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7dbd63c-a2e9-42c5-ac03-3545f6c8f78e None admin] RBAC: Authorizing `identity:get_user(user_id=bb0c79040c314d73be1f0167f3136ef4)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:17.561868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7dbd63c-a2e9-42c5-ac03-3545f6c8f78e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:17.563835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 168/333] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:17 2022] GET /identity/v3/users/bb0c79040c314d73be1f0167f3136ef4 => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:17.570093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:17.698249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faabe07b-18f7-4e61-865d-dbaadb48914a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:17.745252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d93f79a-9d6d-4088-a62f-4a712899509d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:17.746098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d93f79a-9d6d-4088-a62f-4a712899509d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:17.746297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d93f79a-9d6d-4088-a62f-4a712899509d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:17.746576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d93f79a-9d6d-4088-a62f-4a712899509d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:17.747302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d93f79a-9d6d-4088-a62f-4a712899509d None admin] RBAC: Adding query filter params (role.id=b0e7999c9d0d472195e1c5dc6f4d4978, user.id=bb0c79040c314d73be1f0167f3136ef4, scope.system=all) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:17.747612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d93f79a-9d6d-4088-a62f-4a712899509d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:17.749075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d93f79a-9d6d-4088-a62f-4a712899509d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:17.759499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:17.769356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 166/334] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Sep 15 13:56:17 2022] GET /identity/v3/role_assignments?user.id=bb0c79040c314d73be1f0167f3136ef4&scope.system=all&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 510 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:19.072241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fcf325-828e-4c81-8c0e-8c7b5d33703e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:19.072772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fcf325-828e-4c81-8c0e-8c7b5d33703e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:19.072772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fcf325-828e-4c81-8c0e-8c7b5d33703e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:19.074459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 169/335] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:19.086885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663bba9-b2b9-49d0-b5a5-7713fb1b756e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:19.087346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663bba9-b2b9-49d0-b5a5-7713fb1b756e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:19.087421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663bba9-b2b9-49d0-b5a5-7713fb1b756e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:19.108019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d663bba9-b2b9-49d0-b5a5-7713fb1b756e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:19.109202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d663bba9-b2b9-49d0-b5a5-7713fb1b756e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:19.203126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 167/336] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:19.495068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddae849-9f05-4f4e-b5d5-c6dd71ad6ff6 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:19.495421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddae849-9f05-4f4e-b5d5-c6dd71ad6ff6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:19.495619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddae849-9f05-4f4e-b5d5-c6dd71ad6ff6 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:19.496987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 170/337] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:19.502504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684c8067-e2fb-4bae-8adc-15e872a4edff None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:19.503058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684c8067-e2fb-4bae-8adc-15e872a4edff None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:19.503581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684c8067-e2fb-4bae-8adc-15e872a4edff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:19.522746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-684c8067-e2fb-4bae-8adc-15e872a4edff None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:19.524005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-684c8067-e2fb-4bae-8adc-15e872a4edff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:19.607317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 168/338] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:19.721170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddae849-9f05-4f4e-b5d5-c6dd71ad6ff6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:19.833067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:19.860396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddae849-9f05-4f4e-b5d5-c6dd71ad6ff6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:19.906035 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c924e020-f164-45d6-965e-e32663d341c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:19.906684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924e020-f164-45d6-965e-e32663d341c1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:19.906962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924e020-f164-45d6-965e-e32663d341c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:19.907235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924e020-f164-45d6-965e-e32663d341c1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:19.921228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c924e020-f164-45d6-965e-e32663d341c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-c924e020-f164-45d6-965e-e32663d341c1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:19.926409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:19.927387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:19.928669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 171/339] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:19.935551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684c8067-e2fb-4bae-8adc-15e872a4edff None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:19.999816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:20.048901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aeeff8-c01f-4cca-9740-99843eec29fa None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.049209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aeeff8-c01f-4cca-9740-99843eec29fa None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.049454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aeeff8-c01f-4cca-9740-99843eec29fa None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.052243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 172/340] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:20.052978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684c8067-e2fb-4bae-8adc-15e872a4edff None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:20.066489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f013ca7-3cf9-42eb-9675-f83110e8eea2 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.066823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f013ca7-3cf9-42eb-9675-f83110e8eea2 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.067027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f013ca7-3cf9-42eb-9675-f83110e8eea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.096018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8f013ca7-3cf9-42eb-9675-f83110e8eea2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:20.101692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8f013ca7-3cf9-42eb-9675-f83110e8eea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:20.137780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:20.138398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.139123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.139544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.140502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:20.140949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:20.142643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:20.162321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 169/341] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:20.168849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:20.209583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 173/342] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:20.210848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f013ca7-3cf9-42eb-9675-f83110e8eea2 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:20.354549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:20.361689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a13779-b2f2-4e41-9dc8-41bbf84040ee None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:20.393556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:20.405538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f013ca7-3cf9-42eb-9675-f83110e8eea2 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:20.431145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c0c43e-a67a-4598-81dd-e45f0c796ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:20.431786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c0c43e-a67a-4598-81dd-e45f0c796ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.432070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c0c43e-a67a-4598-81dd-e45f0c796ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.432349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c0c43e-a67a-4598-81dd-e45f0c796ae6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.475250 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a09662-65c3-4a19-8709-259cb23b6894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:20.475920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a09662-65c3-4a19-8709-259cb23b6894 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.476204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a09662-65c3-4a19-8709-259cb23b6894 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.476481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a09662-65c3-4a19-8709-259cb23b6894 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.497968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a09662-65c3-4a19-8709-259cb23b6894 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-36a09662-65c3-4a19-8709-259cb23b6894 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:20.508069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:20.508966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:20.509768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 174/343] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:20.516289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f490c76d-ebfc-4d51-936a-a1d0e64875e4 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.516575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f490c76d-ebfc-4d51-936a-a1d0e64875e4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.516850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f490c76d-ebfc-4d51-936a-a1d0e64875e4 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.518887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 175/344] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:20 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:20.525729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce316e6b-b92f-4702-b7f2-d1efc89b5e59 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.526016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce316e6b-b92f-4702-b7f2-d1efc89b5e59 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.526293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce316e6b-b92f-4702-b7f2-d1efc89b5e59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.556441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ce316e6b-b92f-4702-b7f2-d1efc89b5e59 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:20.557632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ce316e6b-b92f-4702-b7f2-d1efc89b5e59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:20.558566 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c0c43e-a67a-4598-81dd-e45f0c796ae6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:20.560082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c0c43e-a67a-4598-81dd-e45f0c796ae6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:20.648789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 176/345] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:20.649580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce316e6b-b92f-4702-b7f2-d1efc89b5e59 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:20.669951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 170/346] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 502 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:20.671135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c0c43e-a67a-4598-81dd-e45f0c796ae6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:20.696809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:20.736365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:20.786292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce316e6b-b92f-4702-b7f2-d1efc89b5e59 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:20.833388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:20.834405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.834700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.834976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.835735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:20.836050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:20.838067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:20.848290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 177/347] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:20.849545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:20.868042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c0c43e-a67a-4598-81dd-e45f0c796ae6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:20.944477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b51674-beb4-4647-a223-e7b312f98058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:20.945033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b51674-beb4-4647-a223-e7b312f98058 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:20.945313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b51674-beb4-4647-a223-e7b312f98058 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:20.945586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b51674-beb4-4647-a223-e7b312f98058 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:20.971196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b51674-beb4-4647-a223-e7b312f98058 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-a2b51674-beb4-4647-a223-e7b312f98058 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:20.980385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:20.981363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 15 13:56:20.982153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:20.982746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 171/348] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:20 2022] GET /identity/v3/roles/reader => generated 84 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:20.983291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b51674-beb4-4647-a223-e7b312f98058 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.045003 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302ed46d-86ef-48eb-aab7-446cf457e26d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.074183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:21.086531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624a7586-b529-48b2-a036-bfd479616b03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.087450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a7586-b529-48b2-a036-bfd479616b03 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.087959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a7586-b529-48b2-a036-bfd479616b03 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.088465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a7586-b529-48b2-a036-bfd479616b03 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.089848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624a7586-b529-48b2-a036-bfd479616b03 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:21.091642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624a7586-b529-48b2-a036-bfd479616b03 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:21.093932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 178/349] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:20 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:21.094884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624a7586-b529-48b2-a036-bfd479616b03 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.103276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:21.131095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b51674-beb4-4647-a223-e7b312f98058 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.173129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d19bc3-3c4d-441f-a82d-688fbe47925c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.174486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d19bc3-3c4d-441f-a82d-688fbe47925c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.174977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d19bc3-3c4d-441f-a82d-688fbe47925c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.176260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d19bc3-3c4d-441f-a82d-688fbe47925c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.229504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624a7586-b529-48b2-a036-bfd479616b03 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.264043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d19bc3-3c4d-441f-a82d-688fbe47925c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:21.265615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d19bc3-3c4d-441f-a82d-688fbe47925c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:21.275787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.276736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.277213 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.277699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.279101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:21.279615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:21.281323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:21.292797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 179/350] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:21.293890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.371034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:21.380375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:21.400304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 172/351] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:21.401162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d19bc3-3c4d-441f-a82d-688fbe47925c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.442366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8043a4-00e5-4fcb-abf0-956179120c46 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.490871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f796c96-d9a6-4356-a253-0f5533450fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.491444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f796c96-d9a6-4356-a253-0f5533450fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.491743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f796c96-d9a6-4356-a253-0f5533450fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.492280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f796c96-d9a6-4356-a253-0f5533450fc5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.506044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f796c96-d9a6-4356-a253-0f5533450fc5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-5f796c96-d9a6-4356-a253-0f5533450fc5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:21.513071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:21.513960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:21.514738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 180/352] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:21.515399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d19bc3-3c4d-441f-a82d-688fbe47925c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.515470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f796c96-d9a6-4356-a253-0f5533450fc5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.561077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e801dc-12c9-4910-b715-ec83ac693614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.562517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e801dc-12c9-4910-b715-ec83ac693614 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.562814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e801dc-12c9-4910-b715-ec83ac693614 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.563092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e801dc-12c9-4910-b715-ec83ac693614 None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.572771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e801dc-12c9-4910-b715-ec83ac693614 None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:21.574279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e801dc-12c9-4910-b715-ec83ac693614 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:21.576435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 173/353] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:21 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:21.577284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e801dc-12c9-4910-b715-ec83ac693614 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.634485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:21.647236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f796c96-d9a6-4356-a253-0f5533450fc5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.669245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:21.697285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1692c503-09b7-4986-b2ab-90d99977cbde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.698422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1692c503-09b7-4986-b2ab-90d99977cbde None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.698422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1692c503-09b7-4986-b2ab-90d99977cbde None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.698636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1692c503-09b7-4986-b2ab-90d99977cbde None admin] PATH_INFO: `/v3/users/63a404f4799147c2b07ac694a13bd48b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.699667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1692c503-09b7-4986-b2ab-90d99977cbde None admin] RBAC: Authorizing `identity:get_user(user_id=63a404f4799147c2b07ac694a13bd48b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:21.702289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1692c503-09b7-4986-b2ab-90d99977cbde None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:21.702951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 181/354] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:56:21 2022] GET /identity/v3/users/63a404f4799147c2b07ac694a13bd48b => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:21.704878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca885d2d-5791-4bf3-8a04-e7932cb854de None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.705162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca885d2d-5791-4bf3-8a04-e7932cb854de None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.705431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca885d2d-5791-4bf3-8a04-e7932cb854de None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.706926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 182/355] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:21.707965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca885d2d-5791-4bf3-8a04-e7932cb854de None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.725658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e801dc-12c9-4910-b715-ec83ac693614 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.768751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbc0415-50eb-44a7-a269-f3ad9711d3e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.769851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbc0415-50eb-44a7-a269-f3ad9711d3e6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.770268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbc0415-50eb-44a7-a269-f3ad9711d3e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.770554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbc0415-50eb-44a7-a269-f3ad9711d3e6 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.772178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbc0415-50eb-44a7-a269-f3ad9711d3e6 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:21.773637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbc0415-50eb-44a7-a269-f3ad9711d3e6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:21.777076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 174/356] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:21 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:21.777890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbc0415-50eb-44a7-a269-f3ad9711d3e6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.850238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca885d2d-5791-4bf3-8a04-e7932cb854de None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.898570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.899641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.900093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.900533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.901210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:21.901512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:21.902901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:21.913439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 183/357] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:21.914283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:21.916777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbc0415-50eb-44a7-a269-f3ad9711d3e6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:21.931344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:21.959457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b2e768-c783-403a-9bdb-a01794605523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:21.960159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b2e768-c783-403a-9bdb-a01794605523 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.960609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b2e768-c783-403a-9bdb-a01794605523 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.960877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b2e768-c783-403a-9bdb-a01794605523 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.961606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b2e768-c783-403a-9bdb-a01794605523 None admin] RBAC: Adding query filter params (group.id=81ecc4767f864a7e9c5d7b5e405f5d22, role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:21.961916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b2e768-c783-403a-9bdb-a01794605523 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:21.963336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b2e768-c783-403a-9bdb-a01794605523 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:21.968992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 175/358] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:21 2022] GET /identity/v3/role_assignments?group.id=81ecc4767f864a7e9c5d7b5e405f5d22&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 260 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:21.971142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4aff62-871e-4009-a096-7fcb3e42efdc None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:21.971704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4aff62-871e-4009-a096-7fcb3e42efdc None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:21.972682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4aff62-871e-4009-a096-7fcb3e42efdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:21.977895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:21.993042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-fc4aff62-871e-4009-a096-7fcb3e42efdc None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:21.994278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fc4aff62-871e-4009-a096-7fcb3e42efdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:22.077357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81283866-5d60-4a51-a215-5995ab534b5b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:22.086050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 176/359] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:22.086050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4aff62-871e-4009-a096-7fcb3e42efdc None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:22.119591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:22.120535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:22.121090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:22.121521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:22.122428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] RBAC: Adding query filter params (role.id=9950301e6a2d4ba594bc506ec017aad0, user.id=63a404f4799147c2b07ac694a13bd48b, scope.system=all) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:22.122906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:22.124550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:22.141387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 184/360] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Sep 15 13:56:21 2022] GET /identity/v3/role_assignments?user.id=63a404f4799147c2b07ac694a13bd48b&scope.system=all&role.id=9950301e6a2d4ba594bc506ec017aad0 => generated 510 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:22.142592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:22.245944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4aff62-871e-4009-a096-7fcb3e42efdc None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:22.250209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:22.301671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99142583-5a0a-4548-bdcf-a29e3b3b9235 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:22.302179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a4aca3-ddcb-44cc-9bd1-cbf774a211ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:22.302493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a4aca3-ddcb-44cc-9bd1-cbf774a211ec None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:22.302773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a4aca3-ddcb-44cc-9bd1-cbf774a211ec None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:22.303045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a4aca3-ddcb-44cc-9bd1-cbf774a211ec None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:22.304211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a4aca3-ddcb-44cc-9bd1-cbf774a211ec None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:22.307164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:22.307300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a4aca3-ddcb-44cc-9bd1-cbf774a211ec None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:22.309324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 177/361] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:22 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:22.310772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a410147f-9720-4f51-a975-d9b4e69c9f14 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:22.311194 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a410147f-9720-4f51-a975-d9b4e69c9f14 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:22.311739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a410147f-9720-4f51-a975-d9b4e69c9f14 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:22.315517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 178/362] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:22.326404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c31512-0233-4a04-a69e-4f372b6344db None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:22.326917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c31512-0233-4a04-a69e-4f372b6344db None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:22.327350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c31512-0233-4a04-a69e-4f372b6344db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:22.352810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-39c31512-0233-4a04-a69e-4f372b6344db None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:22.354260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-39c31512-0233-4a04-a69e-4f372b6344db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:22.381284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547b4699-7199-4f1d-b07c-f3b7fd5b9136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:22.382162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b4699-7199-4f1d-b07c-f3b7fd5b9136 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:22.382593 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b4699-7199-4f1d-b07c-f3b7fd5b9136 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:22.383035 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b4699-7199-4f1d-b07c-f3b7fd5b9136 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:22.442248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 179/363] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:22.449015 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c31512-0233-4a04-a69e-4f372b6344db None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:22.477854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547b4699-7199-4f1d-b07c-f3b7fd5b9136 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:22.479636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547b4699-7199-4f1d-b07c-f3b7fd5b9136 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:22.583905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c31512-0233-4a04-a69e-4f372b6344db None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:22.585558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:22.615850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 185/364] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:22.616948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547b4699-7199-4f1d-b07c-f3b7fd5b9136 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:22.621156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:22.642660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:22.643548 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:22.644528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:22.645082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:22.646986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] RBAC: Adding query filter params (group.id=81ecc4767f864a7e9c5d7b5e405f5d22, role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:22.647508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:22.649178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:22.656521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 180/365] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:22 2022] GET /identity/v3/role_assignments?group.id=81ecc4767f864a7e9c5d7b5e405f5d22&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 260 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:22.657572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:22.760576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547b4699-7199-4f1d-b07c-f3b7fd5b9136 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:22.780707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea8234-d3b2-4ad6-b4bc-32cf284b84a8 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:22.810087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f3dd38-145b-490d-bf9d-f9a033ef9743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:22.810795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f3dd38-145b-490d-bf9d-f9a033ef9743 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:22.811281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f3dd38-145b-490d-bf9d-f9a033ef9743 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:22.811814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f3dd38-145b-490d-bf9d-f9a033ef9743 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:22.821445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb66662-cadb-42db-8293-dbebda6b46cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:22.822423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb66662-cadb-42db-8293-dbebda6b46cc None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:22.822877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb66662-cadb-42db-8293-dbebda6b46cc None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:22.823323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb66662-cadb-42db-8293-dbebda6b46cc None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:22.825734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f3dd38-145b-490d-bf9d-f9a033ef9743 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:22.827718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb66662-cadb-42db-8293-dbebda6b46cc None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:22.829456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb66662-cadb-42db-8293-dbebda6b46cc None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-38f3dd38-145b-490d-bf9d-f9a033ef9743 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:22.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:22.833393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:22.834377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:22.835028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 181/366] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:22 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:22.836605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 186/367] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:22 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:22.839917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb66662-cadb-42db-8293-dbebda6b46cc None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:22.844830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f3dd38-145b-490d-bf9d-f9a033ef9743 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:22.886146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:22.917049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:22.970730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f3dd38-145b-490d-bf9d-f9a033ef9743 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:22.983536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb66662-cadb-42db-8293-dbebda6b46cc None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:23.015425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa105396-b179-4150-b888-b638cad15dec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.016092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa105396-b179-4150-b888-b638cad15dec None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.016376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa105396-b179-4150-b888-b638cad15dec None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.016652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa105396-b179-4150-b888-b638cad15dec None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.017330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa105396-b179-4150-b888-b638cad15dec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:23.017634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa105396-b179-4150-b888-b638cad15dec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:23.019085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa105396-b179-4150-b888-b638cad15dec None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:23.028922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 187/368] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:23.029355 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7d3846-ec7a-41b9-99f3-4761842ef3b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.029743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa105396-b179-4150-b888-b638cad15dec None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:23.030471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7d3846-ec7a-41b9-99f3-4761842ef3b6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.030933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7d3846-ec7a-41b9-99f3-4761842ef3b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.031393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7d3846-ec7a-41b9-99f3-4761842ef3b6 None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.036360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7d3846-ec7a-41b9-99f3-4761842ef3b6 None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:23.038490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7d3846-ec7a-41b9-99f3-4761842ef3b6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:23.041665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 182/369] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:22 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:23.042686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7d3846-ec7a-41b9-99f3-4761842ef3b6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:23.171580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7d3846-ec7a-41b9-99f3-4761842ef3b6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:23.173632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:23.184139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa105396-b179-4150-b888-b638cad15dec None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:23.216810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:23.227321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb410f9b-6036-44ec-bb18-5d3ec4240f8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.228332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb410f9b-6036-44ec-bb18-5d3ec4240f8d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.228757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb410f9b-6036-44ec-bb18-5d3ec4240f8d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.229193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb410f9b-6036-44ec-bb18-5d3ec4240f8d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.229516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dc5343-f1bb-42b1-8b96-2c30a39e9423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.230516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc5343-f1bb-42b1-8b96-2c30a39e9423 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.230997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc5343-f1bb-42b1-8b96-2c30a39e9423 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.231463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dc5343-f1bb-42b1-8b96-2c30a39e9423 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.244311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dc5343-f1bb-42b1-8b96-2c30a39e9423 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-07dc5343-f1bb-42b1-8b96-2c30a39e9423 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:23.251123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:23.252056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:23.252842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:23.255504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 188/370] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:56:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:23.260729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dc5343-f1bb-42b1-8b96-2c30a39e9423 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:23.312176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb410f9b-6036-44ec-bb18-5d3ec4240f8d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:23.313653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb410f9b-6036-44ec-bb18-5d3ec4240f8d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:23.380485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dc5343-f1bb-42b1-8b96-2c30a39e9423 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:23.426659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.427611 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.428587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.429076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.430256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] RBAC: Adding query filter params (group.id=81ecc4767f864a7e9c5d7b5e405f5d22, role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:23.430920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:23.433238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:23.440283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 189/371] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:23 2022] GET /identity/v3/role_assignments?group.id=81ecc4767f864a7e9c5d7b5e405f5d22&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 260 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:23.441444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:23.443830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 183/372] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:23.454507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb410f9b-6036-44ec-bb18-5d3ec4240f8d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:23.456199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:23.549818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:23.619510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6992add-0ff0-4e4f-9c5b-75b1410db554 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:23.654923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb410f9b-6036-44ec-bb18-5d3ec4240f8d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:23.665067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.666364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.667156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.667634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.668797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:23.669306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:23.671775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:23.687244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 190/373] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:23.692987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:23.705851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d6fcaa-340e-4d0f-a5e9-1486b70c739b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.706655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6fcaa-340e-4d0f-a5e9-1486b70c739b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.706936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6fcaa-340e-4d0f-a5e9-1486b70c739b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.707208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6fcaa-340e-4d0f-a5e9-1486b70c739b None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.712545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d6fcaa-340e-4d0f-a5e9-1486b70c739b None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:23.714033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d6fcaa-340e-4d0f-a5e9-1486b70c739b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:23.716109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 184/374] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:23 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:23.723002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d6fcaa-340e-4d0f-a5e9-1486b70c739b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:23.807984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:23.824206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55f9c09-641d-49a1-8c46-18b60b1a66d9 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:23.840256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:23.849774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d6fcaa-340e-4d0f-a5e9-1486b70c739b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:23.864155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7370fc17-8851-464b-83fd-214a1ab340f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.864864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7370fc17-8851-464b-83fd-214a1ab340f1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.865216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7370fc17-8851-464b-83fd-214a1ab340f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.865575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7370fc17-8851-464b-83fd-214a1ab340f1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.888746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3250665-c6bb-443e-83cb-1a1eb19d6f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:23.889380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3250665-c6bb-443e-83cb-1a1eb19d6f42 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:23.889657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3250665-c6bb-443e-83cb-1a1eb19d6f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:23.889926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3250665-c6bb-443e-83cb-1a1eb19d6f42 None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:23.891028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3250665-c6bb-443e-83cb-1a1eb19d6f42 None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:23.892487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3250665-c6bb-443e-83cb-1a1eb19d6f42 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:23.894877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 185/375] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:23 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:23.901107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3250665-c6bb-443e-83cb-1a1eb19d6f42 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:23.951466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7370fc17-8851-464b-83fd-214a1ab340f1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:23.953348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7370fc17-8851-464b-83fd-214a1ab340f1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:24.026417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3250665-c6bb-443e-83cb-1a1eb19d6f42 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:24.080611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:24.081861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:24.082289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:24.082817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:24.083713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] RBAC: Adding query filter params (group.id=81ecc4767f864a7e9c5d7b5e405f5d22, role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:24.084085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:24.085584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:24.092947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 186/376] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:23 2022] GET /identity/v3/role_assignments?group.id=81ecc4767f864a7e9c5d7b5e405f5d22&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 260 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:24.110200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 191/377] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:56:24.118328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:24.126921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:24.261692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c980c871-4230-4ee9-b0af-5e9a235f3996 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:24.309461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b75b0ac-5991-4a31-b25c-d97ab164131c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:24.312165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b75b0ac-5991-4a31-b25c-d97ab164131c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:24.312165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b75b0ac-5991-4a31-b25c-d97ab164131c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:24.312165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b75b0ac-5991-4a31-b25c-d97ab164131c None admin] PATH_INFO: `/v3/projects/f973480c94d94371b4cef62cd36d8410` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:24.312165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b75b0ac-5991-4a31-b25c-d97ab164131c None admin] RBAC: Authorizing `identity:get_project(project_id=f973480c94d94371b4cef62cd36d8410)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:24.313246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b75b0ac-5991-4a31-b25c-d97ab164131c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:24.315889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 187/378] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:24 2022] GET /identity/v3/projects/f973480c94d94371b4cef62cd36d8410 => generated 344 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:24.321734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7370fc17-8851-464b-83fd-214a1ab340f1 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:24.339426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:24.459067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7370fc17-8851-464b-83fd-214a1ab340f1 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:24.500959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c481619e-817a-4c6f-850b-ff69a5fa467d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:24.501600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c481619e-817a-4c6f-850b-ff69a5fa467d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:24.501875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c481619e-817a-4c6f-850b-ff69a5fa467d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:24.502141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c481619e-817a-4c6f-850b-ff69a5fa467d None admin] PATH_INFO: `/v3/groups/bb6c07e9b1d7471bbe04d1115a58efa5` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:24.509579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c481619e-817a-4c6f-850b-ff69a5fa467d None admin] RBAC: Authorizing `identity:get_group(group_id=bb6c07e9b1d7471bbe04d1115a58efa5)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:24.511336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c481619e-817a-4c6f-850b-ff69a5fa467d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:24.513707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 192/379] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:24 2022] GET /identity/v3/groups/bb6c07e9b1d7471bbe04d1115a58efa5 => generated 229 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:24.519502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b75b0ac-5991-4a31-b25c-d97ab164131c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:24.616007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:24.654752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b75b0ac-5991-4a31-b25c-d97ab164131c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:24.695747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0827138f-a1d3-4184-9bf0-35bcd96ba0d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:24.696385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827138f-a1d3-4184-9bf0-35bcd96ba0d9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:24.696664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827138f-a1d3-4184-9bf0-35bcd96ba0d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:24.696935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827138f-a1d3-4184-9bf0-35bcd96ba0d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:24.697683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0827138f-a1d3-4184-9bf0-35bcd96ba0d9 None admin] RBAC: Adding query filter params (group.id=bb6c07e9b1d7471bbe04d1115a58efa5, role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=f973480c94d94371b4cef62cd36d8410) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:24.697992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0827138f-a1d3-4184-9bf0-35bcd96ba0d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:24.699459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0827138f-a1d3-4184-9bf0-35bcd96ba0d9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:24.706395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 188/380] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:24 2022] GET /identity/v3/role_assignments?group.id=bb6c07e9b1d7471bbe04d1115a58efa5&scope.project.id=f973480c94d94371b4cef62cd36d8410&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 260 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:24.954313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8b8c4-c465-401c-950b-46c9acb739d8 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:24.954497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8b8c4-c465-401c-950b-46c9acb739d8 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:24.954761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd8b8c4-c465-401c-950b-46c9acb739d8 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:24.957119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 193/381] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:24.962479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248eea3a-dedc-4e3b-bf70-b55acc4421e6 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:24.962764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248eea3a-dedc-4e3b-bf70-b55acc4421e6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:24.963042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248eea3a-dedc-4e3b-bf70-b55acc4421e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:24.986890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-248eea3a-dedc-4e3b-bf70-b55acc4421e6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:24.989134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-248eea3a-dedc-4e3b-bf70-b55acc4421e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:25.070126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 189/382] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:25.476552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd8b8c4-c465-401c-950b-46c9acb739d8 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:25.566375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:25.594964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd8b8c4-c465-401c-950b-46c9acb739d8 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:25.633146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9db52ad-6a7e-4c9f-b546-40ef0aaf3325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:25.633786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9db52ad-6a7e-4c9f-b546-40ef0aaf3325 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:25.634121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9db52ad-6a7e-4c9f-b546-40ef0aaf3325 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:25.634396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9db52ad-6a7e-4c9f-b546-40ef0aaf3325 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:25.645910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9db52ad-6a7e-4c9f-b546-40ef0aaf3325 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-f9db52ad-6a7e-4c9f-b546-40ef0aaf3325 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:25.651093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:25.652041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:25.652926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 194/383] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:25.660693 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248eea3a-dedc-4e3b-bf70-b55acc4421e6 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:25.737247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:25.792892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248eea3a-dedc-4e3b-bf70-b55acc4421e6 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:25.838685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0875d42a-46a3-4f55-a598-fbaf5ca7d481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:25.839253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875d42a-46a3-4f55-a598-fbaf5ca7d481 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:25.839546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875d42a-46a3-4f55-a598-fbaf5ca7d481 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:25.839847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875d42a-46a3-4f55-a598-fbaf5ca7d481 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:25.840520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0875d42a-46a3-4f55-a598-fbaf5ca7d481 None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:25.840826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0875d42a-46a3-4f55-a598-fbaf5ca7d481 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:25.842190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0875d42a-46a3-4f55-a598-fbaf5ca7d481 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:25.853928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 190/384] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:25.859774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9db52ad-6a7e-4c9f-b546-40ef0aaf3325 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:25.896975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff252b-edbc-4781-9b6a-1e8031f5ce40 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:25.897256 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff252b-edbc-4781-9b6a-1e8031f5ce40 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:25.897697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff252b-edbc-4781-9b6a-1e8031f5ce40 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:25.899449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 191/385] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:25.914697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165af04f-e128-41d9-9e88-0142acb8fdd3 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:25.915207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165af04f-e128-41d9-9e88-0142acb8fdd3 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:25.915650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165af04f-e128-41d9-9e88-0142acb8fdd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:25.948946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-165af04f-e128-41d9-9e88-0142acb8fdd3 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:25.950123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-165af04f-e128-41d9-9e88-0142acb8fdd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:26.035091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9db52ad-6a7e-4c9f-b546-40ef0aaf3325 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:26.082108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 192/386] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:26.094784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414f56f0-8bfa-4e73-9ada-20cd65c05afa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:26.095770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f56f0-8bfa-4e73-9ada-20cd65c05afa None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:26.096256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f56f0-8bfa-4e73-9ada-20cd65c05afa None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:26.096731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f56f0-8bfa-4e73-9ada-20cd65c05afa None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:26.098732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414f56f0-8bfa-4e73-9ada-20cd65c05afa None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:26.100422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414f56f0-8bfa-4e73-9ada-20cd65c05afa None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:26.102557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 195/387] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:25 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:26.107881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165af04f-e128-41d9-9e88-0142acb8fdd3 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:26.177255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:26.228813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165af04f-e128-41d9-9e88-0142acb8fdd3 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:26.266535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7550843f-f035-4047-80a7-e1fd8dee6e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:26.267238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7550843f-f035-4047-80a7-e1fd8dee6e26 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:26.267523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7550843f-f035-4047-80a7-e1fd8dee6e26 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:26.267831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7550843f-f035-4047-80a7-e1fd8dee6e26 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:26.269544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7550843f-f035-4047-80a7-e1fd8dee6e26 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:26.270936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7550843f-f035-4047-80a7-e1fd8dee6e26 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:26.273246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 193/388] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:26 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:26.280797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414f56f0-8bfa-4e73-9ada-20cd65c05afa None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:26.289790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:26.441420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414f56f0-8bfa-4e73-9ada-20cd65c05afa None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:26.478780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4110919-731d-46f9-8ecf-eb0a8059c8dd None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:26.479565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4110919-731d-46f9-8ecf-eb0a8059c8dd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:26.480166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4110919-731d-46f9-8ecf-eb0a8059c8dd None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:26.482038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fa5072-ba39-423c-8eb9-1238f7676e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:26.482769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa5072-ba39-423c-8eb9-1238f7676e54 None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:26.482968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa5072-ba39-423c-8eb9-1238f7676e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:26.483166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 194/389] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:26.483267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa5072-ba39-423c-8eb9-1238f7676e54 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/groups/81ecc4767f864a7e9c5d7b5e405f5d22/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:26.489930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fa5072-ba39-423c-8eb9-1238f7676e54 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d187412d8794cd7a7ad12d6f240c67e, group_id=81ecc4767f864a7e9c5d7b5e405f5d22, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:26.490561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10176be3-5e89-4a49-ac6e-197de196faee None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:26.491043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10176be3-5e89-4a49-ac6e-197de196faee None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:26.491382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fa5072-ba39-423c-8eb9-1238f7676e54 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:26.491759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10176be3-5e89-4a49-ac6e-197de196faee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:26.524018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 196/390] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 15 13:56:26 2022] PUT /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/groups/81ecc4767f864a7e9c5d7b5e405f5d22/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:26.524872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fa5072-ba39-423c-8eb9-1238f7676e54 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:26.525255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-10176be3-5e89-4a49-ac6e-197de196faee None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:26.526622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-10176be3-5e89-4a49-ac6e-197de196faee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:26.618436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 195/391] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:26.669777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:26.731941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fa5072-ba39-423c-8eb9-1238f7676e54 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:26.789449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d92e6e9-84ce-456a-b9b3-efccdceeb84d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:26.790111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92e6e9-84ce-456a-b9b3-efccdceeb84d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:26.790394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92e6e9-84ce-456a-b9b3-efccdceeb84d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:26.790669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92e6e9-84ce-456a-b9b3-efccdceeb84d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:26.802199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d92e6e9-84ce-456a-b9b3-efccdceeb84d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-7d92e6e9-84ce-456a-b9b3-efccdceeb84d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:26.807495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:26.808610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:26.809723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 197/392] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:26.817234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10176be3-5e89-4a49-ac6e-197de196faee None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:26.843278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa81789d-8976-4890-b96f-ec4ab229e677 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:26.843655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa81789d-8976-4890-b96f-ec4ab229e677 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:26.844058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa81789d-8976-4890-b96f-ec4ab229e677 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:26.847278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 198/393] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:26.851887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8417aae1-aeab-40ba-bca5-66dd3ac6ea2e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:26.852249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8417aae1-aeab-40ba-bca5-66dd3ac6ea2e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:26.852594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8417aae1-aeab-40ba-bca5-66dd3ac6ea2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:26.883561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8417aae1-aeab-40ba-bca5-66dd3ac6ea2e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:26.885129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8417aae1-aeab-40ba-bca5-66dd3ac6ea2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:26.902362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:26.960937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10176be3-5e89-4a49-ac6e-197de196faee None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:26.961402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 199/394] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:26.962322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8417aae1-aeab-40ba-bca5-66dd3ac6ea2e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.005271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.005824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.006092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.006366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.007036 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:27.007336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:27.008737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:27.035190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 196/395] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:27.041832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.061231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:27.143127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8417aae1-aeab-40ba-bca5-66dd3ac6ea2e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.197050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d9d6c-3d3f-4c38-9137-aca3552606d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.198055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d9d6c-3d3f-4c38-9137-aca3552606d4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.199489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d9d6c-3d3f-4c38-9137-aca3552606d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.200015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d9d6c-3d3f-4c38-9137-aca3552606d4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.221934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832d9d6c-3d3f-4c38-9137-aca3552606d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-832d9d6c-3d3f-4c38-9137-aca3552606d4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:27.228254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:27.229163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:27.229948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:27.232378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 200/396] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:27.233786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cf5286-0dd7-4b7b-a4d0-a0414b76befd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.242471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d9d6c-3d3f-4c38-9137-aca3552606d4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.273138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:27.341269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3743b33-57dc-4994-a2b9-70f1a2e6933b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.341914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3743b33-57dc-4994-a2b9-70f1a2e6933b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.342193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3743b33-57dc-4994-a2b9-70f1a2e6933b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.342465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3743b33-57dc-4994-a2b9-70f1a2e6933b None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.343588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3743b33-57dc-4994-a2b9-70f1a2e6933b None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:27.345062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3743b33-57dc-4994-a2b9-70f1a2e6933b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:27.351264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 197/397] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:27 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:27.351264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3743b33-57dc-4994-a2b9-70f1a2e6933b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.376753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d9d6c-3d3f-4c38-9137-aca3552606d4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.391756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:27.414571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.415334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.415480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.415724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.416398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:27.416714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:27.418145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:27.429900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 201/398] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:27.431159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.497677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3743b33-57dc-4994-a2b9-70f1a2e6933b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.539443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926651c9-9734-4824-92bd-21b4233ad096 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.540027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926651c9-9734-4824-92bd-21b4233ad096 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.540308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926651c9-9734-4824-92bd-21b4233ad096 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.540587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926651c9-9734-4824-92bd-21b4233ad096 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.548261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa0c736-850e-4ecd-88c7-37ed10f62c4d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.552707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926651c9-9734-4824-92bd-21b4233ad096 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-926651c9-9734-4824-92bd-21b4233ad096 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:27.557487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:27.558378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:27.559507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 198/399] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:27.560917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926651c9-9734-4824-92bd-21b4233ad096 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.596567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08e6e0-968b-418b-8d99-8b11bd071320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.598742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08e6e0-968b-418b-8d99-8b11bd071320 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.599203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08e6e0-968b-418b-8d99-8b11bd071320 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.599622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08e6e0-968b-418b-8d99-8b11bd071320 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.601206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08e6e0-968b-418b-8d99-8b11bd071320 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:27.603718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08e6e0-968b-418b-8d99-8b11bd071320 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:27.610267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 202/400] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:27 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:27.611324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08e6e0-968b-418b-8d99-8b11bd071320 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.614686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:27.662812 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:27.677758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926651c9-9734-4824-92bd-21b4233ad096 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.716376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62adc51-2798-4ae9-9926-596d7a8a4cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.717112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62adc51-2798-4ae9-9926-596d7a8a4cb6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.717462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62adc51-2798-4ae9-9926-596d7a8a4cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.717807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62adc51-2798-4ae9-9926-596d7a8a4cb6 None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.719432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62adc51-2798-4ae9-9926-596d7a8a4cb6 None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:27.720873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62adc51-2798-4ae9-9926-596d7a8a4cb6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:27.727902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 199/401] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:27 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:27.728787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62adc51-2798-4ae9-9926-596d7a8a4cb6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.729841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08e6e0-968b-418b-8d99-8b11bd071320 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.769166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.770106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.770597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.771099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.772019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:27.772519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:27.774129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:27.784605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 203/402] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:27.785871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.841981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62adc51-2798-4ae9-9926-596d7a8a4cb6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.856151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:27.880090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24182651-d868-4859-a3aa-323805c42175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.880914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24182651-d868-4859-a3aa-323805c42175 None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.881158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24182651-d868-4859-a3aa-323805c42175 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.881523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24182651-d868-4859-a3aa-323805c42175 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/groups/81ecc4767f864a7e9c5d7b5e405f5d22/roles/a0256b6f3d6c4517be88b324fcbc1145` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.883942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24182651-d868-4859-a3aa-323805c42175 None admin] RBAC: Authorizing `identity:create_grant(project_id=3d187412d8794cd7a7ad12d6f240c67e, group_id=81ecc4767f864a7e9c5d7b5e405f5d22, role_id=a0256b6f3d6c4517be88b324fcbc1145)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:27.885397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24182651-d868-4859-a3aa-323805c42175 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:27.898412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4442101b-549d-4c89-989e-923fe81b8c58 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:27.903843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 200/403] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 15 13:56:27 2022] PUT /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e/groups/81ecc4767f864a7e9c5d7b5e405f5d22/roles/a0256b6f3d6c4517be88b324fcbc1145 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:27.903843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24182651-d868-4859-a3aa-323805c42175 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:27.912511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:27.952039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ace52d-11d8-4527-97de-bd3962eaf456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:27.952739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ace52d-11d8-4527-97de-bd3962eaf456 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.953025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ace52d-11d8-4527-97de-bd3962eaf456 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.953294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ace52d-11d8-4527-97de-bd3962eaf456 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.957043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ace52d-11d8-4527-97de-bd3962eaf456 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:27.958895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ace52d-11d8-4527-97de-bd3962eaf456 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:27.961795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 204/404] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:27 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:27.965150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc7261-1736-4571-9a11-b9c3fa91a3fe None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:27.965719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc7261-1736-4571-9a11-b9c3fa91a3fe None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:27.966282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc7261-1736-4571-9a11-b9c3fa91a3fe None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:27.968314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 205/405] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:27.970051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cc7261-1736-4571-9a11-b9c3fa91a3fe None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:28.044000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24182651-d868-4859-a3aa-323805c42175 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:28.086991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d305cd-35f2-448d-a29d-772cae8c2740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:28.087636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d305cd-35f2-448d-a29d-772cae8c2740 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:28.087950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d305cd-35f2-448d-a29d-772cae8c2740 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:28.088222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d305cd-35f2-448d-a29d-772cae8c2740 None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:28.089364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d305cd-35f2-448d-a29d-772cae8c2740 None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:28.090830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d305cd-35f2-448d-a29d-772cae8c2740 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:28.092615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 201/406] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:27 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:28.093646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec535b4d-63de-43bd-9768-7c656737af19 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:28.093930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec535b4d-63de-43bd-9768-7c656737af19 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:28.094202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec535b4d-63de-43bd-9768-7c656737af19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:28.116988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ec535b4d-63de-43bd-9768-7c656737af19 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:28.118185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ec535b4d-63de-43bd-9768-7c656737af19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:28.129892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cc7261-1736-4571-9a11-b9c3fa91a3fe None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:28.157384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:28.178353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389ececc-bafb-4b83-b2a6-5f277ac0c37f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:28.179346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389ececc-bafb-4b83-b2a6-5f277ac0c37f None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:28.179621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389ececc-bafb-4b83-b2a6-5f277ac0c37f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:28.179957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389ececc-bafb-4b83-b2a6-5f277ac0c37f None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/groups/81ecc4767f864a7e9c5d7b5e405f5d22/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:28.181905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389ececc-bafb-4b83-b2a6-5f277ac0c37f None admin] RBAC: Authorizing `identity:create_grant(project_id=6fa314f8793248fb8bbde6daa676e4e7, group_id=81ecc4767f864a7e9c5d7b5e405f5d22, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:28.184157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389ececc-bafb-4b83-b2a6-5f277ac0c37f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:28.198388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 202/407] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:28.199788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec535b4d-63de-43bd-9768-7c656737af19 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:28.206657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 206/408] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 15 13:56:27 2022] PUT /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/groups/81ecc4767f864a7e9c5d7b5e405f5d22/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:28.368906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec535b4d-63de-43bd-9768-7c656737af19 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:28.419023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0b168e-8546-4156-84ff-92f0b5b1b99c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:28.420049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b168e-8546-4156-84ff-92f0b5b1b99c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:28.420471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b168e-8546-4156-84ff-92f0b5b1b99c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:28.420929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b168e-8546-4156-84ff-92f0b5b1b99c None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:28.422237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0b168e-8546-4156-84ff-92f0b5b1b99c None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:28.423885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0b168e-8546-4156-84ff-92f0b5b1b99c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:28.426410 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 203/409] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:28 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:28.431936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389ececc-bafb-4b83-b2a6-5f277ac0c37f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:28.490058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:28.539653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0b168e-8546-4156-84ff-92f0b5b1b99c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:28.582017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389ececc-bafb-4b83-b2a6-5f277ac0c37f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:28.591192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:28.624330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e734e5b-0bdd-4f58-bd04-93ccd6b28901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:28.625375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e734e5b-0bdd-4f58-bd04-93ccd6b28901 None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:28.625903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e734e5b-0bdd-4f58-bd04-93ccd6b28901 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:28.626393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e734e5b-0bdd-4f58-bd04-93ccd6b28901 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/groups/81ecc4767f864a7e9c5d7b5e405f5d22/roles/a0256b6f3d6c4517be88b324fcbc1145` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:28.629144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e734e5b-0bdd-4f58-bd04-93ccd6b28901 None admin] RBAC: Authorizing `identity:create_grant(project_id=6fa314f8793248fb8bbde6daa676e4e7, group_id=81ecc4767f864a7e9c5d7b5e405f5d22, role_id=a0256b6f3d6c4517be88b324fcbc1145)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:28.630964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e734e5b-0bdd-4f58-bd04-93ccd6b28901 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:28.649505 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 207/410] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 15 13:56:28 2022] PUT /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7/groups/81ecc4767f864a7e9c5d7b5e405f5d22/roles/a0256b6f3d6c4517be88b324fcbc1145 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:28.693987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0b168e-8546-4156-84ff-92f0b5b1b99c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:28.750364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae8b7dd-fc13-4cdd-9aa6-2c335ab10705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:28.751020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae8b7dd-fc13-4cdd-9aa6-2c335ab10705 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:28.751302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae8b7dd-fc13-4cdd-9aa6-2c335ab10705 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:28.751578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae8b7dd-fc13-4cdd-9aa6-2c335ab10705 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:28.771502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae8b7dd-fc13-4cdd-9aa6-2c335ab10705 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-6ae8b7dd-fc13-4cdd-9aa6-2c335ab10705 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:28.776533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:28.777466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:28.778368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 204/411] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:56:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:28.783606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e734e5b-0bdd-4f58-bd04-93ccd6b28901 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:28.929060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:28.940153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e734e5b-0bdd-4f58-bd04-93ccd6b28901 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:28.979450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010ba78d-b3ae-443c-a4b2-68c8789e9be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:28.980129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010ba78d-b3ae-443c-a4b2-68c8789e9be1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:28.980409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010ba78d-b3ae-443c-a4b2-68c8789e9be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:28.980677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010ba78d-b3ae-443c-a4b2-68c8789e9be1 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:28.981350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010ba78d-b3ae-443c-a4b2-68c8789e9be1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:28.981649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010ba78d-b3ae-443c-a4b2-68c8789e9be1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:28.983022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010ba78d-b3ae-443c-a4b2-68c8789e9be1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:28.994913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 208/412] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:29.000768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae8b7dd-fc13-4cdd-9aa6-2c335ab10705 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:29.063400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d0b17-2ad4-48f9-b140-da508af950c8 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:29.063746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d0b17-2ad4-48f9-b140-da508af950c8 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:29.063965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700d0b17-2ad4-48f9-b140-da508af950c8 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:29.065292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 209/413] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:29.076685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71b505-f3c8-4c69-98b9-99d61e4e1d3e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:29.076959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71b505-f3c8-4c69-98b9-99d61e4e1d3e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:29.077238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71b505-f3c8-4c69-98b9-99d61e4e1d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:29.115099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6a71b505-f3c8-4c69-98b9-99d61e4e1d3e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:29.117167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6a71b505-f3c8-4c69-98b9-99d61e4e1d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:29.141546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:29.147162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae8b7dd-fc13-4cdd-9aa6-2c335ab10705 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:29.213715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f3ad7-c73d-4527-ba74-561a4cafe7c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:29.214349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f3ad7-c73d-4527-ba74-561a4cafe7c3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:29.214625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f3ad7-c73d-4527-ba74-561a4cafe7c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:29.214897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f3ad7-c73d-4527-ba74-561a4cafe7c3 None admin] PATH_INFO: `/v3/groups/bb6c07e9b1d7471bbe04d1115a58efa5` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:29.216205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1f3ad7-c73d-4527-ba74-561a4cafe7c3 None admin] RBAC: Authorizing `identity:get_group(group_id=bb6c07e9b1d7471bbe04d1115a58efa5)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:29.217634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1f3ad7-c73d-4527-ba74-561a4cafe7c3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:29.220408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 205/414] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:28 2022] GET /identity/v3/groups/bb6c07e9b1d7471bbe04d1115a58efa5 => generated 229 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:29.229482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f3ad7-c73d-4527-ba74-561a4cafe7c3 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:29.253067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 210/415] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:29.390035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f3ad7-c73d-4527-ba74-561a4cafe7c3 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:29.425209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b969971-c168-4292-bb2c-116ae47ba980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:29.426097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b969971-c168-4292-bb2c-116ae47ba980 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:29.426375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b969971-c168-4292-bb2c-116ae47ba980 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:29.426699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b969971-c168-4292-bb2c-116ae47ba980 None admin] PATH_INFO: `/v3/projects/f973480c94d94371b4cef62cd36d8410` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:29.427903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b969971-c168-4292-bb2c-116ae47ba980 None admin] RBAC: Authorizing `identity:get_project(project_id=f973480c94d94371b4cef62cd36d8410)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:29.429452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b969971-c168-4292-bb2c-116ae47ba980 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:29.431369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 206/416] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:29 2022] GET /identity/v3/projects/f973480c94d94371b4cef62cd36d8410 => generated 344 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:29.437020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a71b505-f3c8-4c69-98b9-99d61e4e1d3e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:29.550344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:29.565876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b969971-c168-4292-bb2c-116ae47ba980 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:29.572292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a71b505-f3c8-4c69-98b9-99d61e4e1d3e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:29.628062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:29.645978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06ecd23-770f-4791-b188-60a14181bc5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:29.647193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06ecd23-770f-4791-b188-60a14181bc5d None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:29.647637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06ecd23-770f-4791-b188-60a14181bc5d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:29.648068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06ecd23-770f-4791-b188-60a14181bc5d None admin] PATH_INFO: `/v3/projects/f973480c94d94371b4cef62cd36d8410/groups/bb6c07e9b1d7471bbe04d1115a58efa5/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:29.653020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06ecd23-770f-4791-b188-60a14181bc5d None admin] RBAC: Authorizing `identity:create_grant(project_id=f973480c94d94371b4cef62cd36d8410, group_id=bb6c07e9b1d7471bbe04d1115a58efa5, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:29.654629 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06ecd23-770f-4791-b188-60a14181bc5d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:29.686247 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 211/417] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 15 13:56:29 2022] PUT /identity/v3/projects/f973480c94d94371b4cef62cd36d8410/groups/bb6c07e9b1d7471bbe04d1115a58efa5/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:29.705128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b969971-c168-4292-bb2c-116ae47ba980 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:29.741912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb79e9e-62dd-4b97-b2d7-8688cdc9aa02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:29.742560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb79e9e-62dd-4b97-b2d7-8688cdc9aa02 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:29.742843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb79e9e-62dd-4b97-b2d7-8688cdc9aa02 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:29.743120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb79e9e-62dd-4b97-b2d7-8688cdc9aa02 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:29.755277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb79e9e-62dd-4b97-b2d7-8688cdc9aa02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-8cb79e9e-62dd-4b97-b2d7-8688cdc9aa02 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:29.761464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:29.762543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:29.763373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 207/418] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:29.768740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06ecd23-770f-4791-b188-60a14181bc5d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:29.906381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06ecd23-770f-4791-b188-60a14181bc5d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:29.949496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:29.950434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:29.950963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:29.951429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:29.952298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] RBAC: Adding query filter params (name=member) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:29.952793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:29.954347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:29.963980 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 212/419] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:29.969327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb79e9e-62dd-4b97-b2d7-8688cdc9aa02 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:30.124320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:30.132938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb79e9e-62dd-4b97-b2d7-8688cdc9aa02 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:30.175987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c3686b-e021-4af5-abcc-c4f61b15005d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:30.176540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3686b-e021-4af5-abcc-c4f61b15005d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:30.176817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3686b-e021-4af5-abcc-c4f61b15005d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:30.177089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3686b-e021-4af5-abcc-c4f61b15005d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:30.251922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c3686b-e021-4af5-abcc-c4f61b15005d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:30.253558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c3686b-e021-4af5-abcc-c4f61b15005d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:30.351556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:30.359883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 208/420] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:56:30.365879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:30.392120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:30.501164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d603ec0-9625-4693-89c5-810811524562 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:30.546644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2ee621-84e0-4e2e-b1dc-147ffa38fbe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:30.547271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2ee621-84e0-4e2e-b1dc-147ffa38fbe4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:30.547548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2ee621-84e0-4e2e-b1dc-147ffa38fbe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:30.547849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2ee621-84e0-4e2e-b1dc-147ffa38fbe4 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:30.552587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2ee621-84e0-4e2e-b1dc-147ffa38fbe4 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:30.554162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2ee621-84e0-4e2e-b1dc-147ffa38fbe4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:30.556240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 213/421] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:30 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:30.561662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c3686b-e021-4af5-abcc-c4f61b15005d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:30.684562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c3686b-e021-4af5-abcc-c4f61b15005d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:30.710875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fff08d-62ab-443a-94de-f2b638874fc1 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:30.711376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fff08d-62ab-443a-94de-f2b638874fc1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:30.711450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fff08d-62ab-443a-94de-f2b638874fc1 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:30.713046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 214/422] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:30.720622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb1164-6e74-4e34-9b32-a334fa87104b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:30.720906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb1164-6e74-4e34-9b32-a334fa87104b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:30.721185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb1164-6e74-4e34-9b32-a334fa87104b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:30.721515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efca88b-f0ba-498c-86d2-9515e936410b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:30.722867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efca88b-f0ba-498c-86d2-9515e936410b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:30.723246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efca88b-f0ba-498c-86d2-9515e936410b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:30.723528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efca88b-f0ba-498c-86d2-9515e936410b None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:30.724696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efca88b-f0ba-498c-86d2-9515e936410b None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:30.726122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efca88b-f0ba-498c-86d2-9515e936410b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:30.728036 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 209/423] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:30 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:30.736170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efca88b-f0ba-498c-86d2-9515e936410b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:30.757679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-debb1164-6e74-4e34-9b32-a334fa87104b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:30.758798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-debb1164-6e74-4e34-9b32-a334fa87104b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:30.833646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:30.910669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efca88b-f0ba-498c-86d2-9515e936410b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:30.913626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 215/424] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:30.921117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8ddc08-b62f-4f84-bb58-45774116d4df None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:30.921637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8ddc08-b62f-4f84-bb58-45774116d4df None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:30.922055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8ddc08-b62f-4f84-bb58-45774116d4df None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:30.923487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 216/425] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:30.934909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa58d4-df89-400b-a54e-23d87b90f850 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:30.935373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa58d4-df89-400b-a54e-23d87b90f850 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:30.935847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa58d4-df89-400b-a54e-23d87b90f850 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:30.955067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:30.969471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c3fa58d4-df89-400b-a54e-23d87b90f850 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:30.971563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c3fa58d4-df89-400b-a54e-23d87b90f850 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:30.980516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:30.981390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:30.981865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:30.982330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:30.983266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] RBAC: Adding query filter params (group.id=81ecc4767f864a7e9c5d7b5e405f5d22, role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:30.983798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:30.985441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:31.004840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 210/426] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:30 2022] GET /identity/v3/role_assignments?group.id=81ecc4767f864a7e9c5d7b5e405f5d22&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 611 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:31.078503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 217/427] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:31.200224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:31.324715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad97dcd-269c-4f18-8d8d-4d90c2492936 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:31.354276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:31.363281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4173e016-04d4-4d97-b8df-0cde3f74edd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:31.363950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4173e016-04d4-4d97-b8df-0cde3f74edd1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:31.364643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4173e016-04d4-4d97-b8df-0cde3f74edd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:31.364918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4173e016-04d4-4d97-b8df-0cde3f74edd1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:31.376390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4173e016-04d4-4d97-b8df-0cde3f74edd1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-4173e016-04d4-4d97-b8df-0cde3f74edd1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:31.383111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:31.384014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:31.384814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 211/428] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:31.392787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fa58d4-df89-400b-a54e-23d87b90f850 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:31.400334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4173e016-04d4-4d97-b8df-0cde3f74edd1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:31.522089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fa58d4-df89-400b-a54e-23d87b90f850 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:31.523120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4173e016-04d4-4d97-b8df-0cde3f74edd1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:31.564886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:31.565649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:31.565974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:31.566248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:31.566924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:31.567217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:31.568651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:31.570673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee950c2f-0f7f-4683-9944-f408300e8847 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:31.571532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:31.572694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee950c2f-0f7f-4683-9944-f408300e8847 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:31.573164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee950c2f-0f7f-4683-9944-f408300e8847 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:31.573624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee950c2f-0f7f-4683-9944-f408300e8847 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:31.580543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 218/429] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:31.587727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:31.596563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee950c2f-0f7f-4683-9944-f408300e8847 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-ee950c2f-0f7f-4683-9944-f408300e8847 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:31.603958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:31.604880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 15 13:56:31.605667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:31.608425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 212/430] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:31.615380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee950c2f-0f7f-4683-9944-f408300e8847 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:31.676810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:31.707384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c482573-ee22-4e62-a3de-a2f7e2b41855 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:31.749969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e20d768-f4e8-4d52-bc3a-035f8b5724e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:31.750745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e20d768-f4e8-4d52-bc3a-035f8b5724e7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:31.751122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e20d768-f4e8-4d52-bc3a-035f8b5724e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:31.751494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e20d768-f4e8-4d52-bc3a-035f8b5724e7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:31.752748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee950c2f-0f7f-4683-9944-f408300e8847 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:31.796762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:31.798116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:31.798543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:31.798971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:31.800687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] RBAC: Adding query filter params (name=member) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:31.801393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:31.803283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:31.814396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 213/431] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:56:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:31.822273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:31.841968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:31.847567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e20d768-f4e8-4d52-bc3a-035f8b5724e7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:31.850891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e20d768-f4e8-4d52-bc3a-035f8b5724e7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:31.954063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553674ba-2be8-4db1-b87d-3014cbbc9484 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:31.965811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:31.976307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 219/432] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:31.977717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b280d51b-d8cc-49b9-8524-a74dc6696ad7 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:31.978161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b280d51b-d8cc-49b9-8524-a74dc6696ad7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:31.978437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b280d51b-d8cc-49b9-8524-a74dc6696ad7 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:31.985480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 220/433] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:31 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:31.988444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b280d51b-d8cc-49b9-8524-a74dc6696ad7 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.001110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c140b3-fbaa-4ff9-a8d9-f063126cd18b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.001720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c140b3-fbaa-4ff9-a8d9-f063126cd18b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.002000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c140b3-fbaa-4ff9-a8d9-f063126cd18b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.002271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c140b3-fbaa-4ff9-a8d9-f063126cd18b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.074872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c140b3-fbaa-4ff9-a8d9-f063126cd18b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:32.076374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c140b3-fbaa-4ff9-a8d9-f063126cd18b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:32.095258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:32.101060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b280d51b-d8cc-49b9-8524-a74dc6696ad7 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:32.137166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403361d-e836-4cf4-b278-5ad7616283e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.137789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403361d-e836-4cf4-b278-5ad7616283e1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403361d-e836-4cf4-b278-5ad7616283e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.138338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403361d-e836-4cf4-b278-5ad7616283e1 None admin] PATH_INFO: `/v3/projects/3d187412d8794cd7a7ad12d6f240c67e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.139457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9403361d-e836-4cf4-b278-5ad7616283e1 None admin] RBAC: Authorizing `identity:get_project(project_id=3d187412d8794cd7a7ad12d6f240c67e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:32.140925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9403361d-e836-4cf4-b278-5ad7616283e1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:32.147530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 221/434] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:31 2022] GET /identity/v3/projects/3d187412d8794cd7a7ad12d6f240c67e => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:32.147530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6e774-2a5f-4c30-b6bd-1a4df8891527 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.147530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6e774-2a5f-4c30-b6bd-1a4df8891527 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.147530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6e774-2a5f-4c30-b6bd-1a4df8891527 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.164228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-46d6e774-2a5f-4c30-b6bd-1a4df8891527 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:32.165416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-46d6e774-2a5f-4c30-b6bd-1a4df8891527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:32.187091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 214/435] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:56:32.188000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c140b3-fbaa-4ff9-a8d9-f063126cd18b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.200986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:32.245802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 222/436] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:32.247371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d6e774-2a5f-4c30-b6bd-1a4df8891527 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.307712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c140b3-fbaa-4ff9-a8d9-f063126cd18b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:32.348421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a559c58-204a-4c98-8551-f9ece2be1cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.349123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a559c58-204a-4c98-8551-f9ece2be1cd5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.349417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a559c58-204a-4c98-8551-f9ece2be1cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.349584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a559c58-204a-4c98-8551-f9ece2be1cd5 None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.350660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a559c58-204a-4c98-8551-f9ece2be1cd5 None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:32.352110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a559c58-204a-4c98-8551-f9ece2be1cd5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:32.358970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 215/437] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:32.358970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c1121-0343-4f5f-a906-cf1bc892456b None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.358970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c1121-0343-4f5f-a906-cf1bc892456b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.358970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c1121-0343-4f5f-a906-cf1bc892456b None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.358970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 216/438] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:32.360553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c1121-0343-4f5f-a906-cf1bc892456b None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.369287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:32.375975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d6e774-2a5f-4c30-b6bd-1a4df8891527 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:32.413004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805209fa-5b82-4ab0-ba4f-7b5fc57aaa1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.413634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805209fa-5b82-4ab0-ba4f-7b5fc57aaa1a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.413916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805209fa-5b82-4ab0-ba4f-7b5fc57aaa1a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.414223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805209fa-5b82-4ab0-ba4f-7b5fc57aaa1a None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.415293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805209fa-5b82-4ab0-ba4f-7b5fc57aaa1a None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:32.416762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805209fa-5b82-4ab0-ba4f-7b5fc57aaa1a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:32.423091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 223/439] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:32.423091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d8b72f-dcc9-4d47-af41-ab96e67dba0d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.423091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d8b72f-dcc9-4d47-af41-ab96e67dba0d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.423091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d8b72f-dcc9-4d47-af41-ab96e67dba0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.438855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e9d8b72f-dcc9-4d47-af41-ab96e67dba0d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:32.440087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e9d8b72f-dcc9-4d47-af41-ab96e67dba0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:32.451569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:32.479835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c1121-0343-4f5f-a906-cf1bc892456b None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:32.512158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 224/440] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:32.513256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d8b72f-dcc9-4d47-af41-ab96e67dba0d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.520952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.521558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.521840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.522113 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.522836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] RBAC: Adding query filter params (group.id=81ecc4767f864a7e9c5d7b5e405f5d22, role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=3d187412d8794cd7a7ad12d6f240c67e) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:32.523145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:32.524653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:32.530885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 217/441] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/role_assignments?group.id=81ecc4767f864a7e9c5d7b5e405f5d22&scope.project.id=3d187412d8794cd7a7ad12d6f240c67e&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 611 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:32.536697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.623244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:32.632936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d8b72f-dcc9-4d47-af41-ab96e67dba0d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:32.676606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e222c3f1-edc9-40f0-8e6b-690f63b3c70c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:32.678733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc049c62-0fa3-4eca-8479-c033d60f19ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.679781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc049c62-0fa3-4eca-8479-c033d60f19ea None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.680273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc049c62-0fa3-4eca-8479-c033d60f19ea None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.680762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc049c62-0fa3-4eca-8479-c033d60f19ea None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.682201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc049c62-0fa3-4eca-8479-c033d60f19ea None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:32.683950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc049c62-0fa3-4eca-8479-c033d60f19ea None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:32.686272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 225/442] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:32.695834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc049c62-0fa3-4eca-8479-c033d60f19ea None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.718283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2181722-4d10-4936-aa1b-386faccf66b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.718914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2181722-4d10-4936-aa1b-386faccf66b5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.719191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2181722-4d10-4936-aa1b-386faccf66b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.719460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2181722-4d10-4936-aa1b-386faccf66b5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.728213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:32.731781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2181722-4d10-4936-aa1b-386faccf66b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-e2181722-4d10-4936-aa1b-386faccf66b5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:32.737173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:32.738205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:32.739023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:32.739627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 218/443] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:32.745142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2181722-4d10-4936-aa1b-386faccf66b5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.837471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc049c62-0fa3-4eca-8479-c033d60f19ea None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:32.866079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2181722-4d10-4936-aa1b-386faccf66b5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:32.874480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.875091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.875370 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.875641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.876399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] RBAC: Adding query filter params (group.id=81ecc4767f864a7e9c5d7b5e405f5d22, role.id=b0e7999c9d0d472195e1c5dc6f4d4978, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:32.876707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:32.878693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:32.885180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 226/444] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/role_assignments?group.id=81ecc4767f864a7e9c5d7b5e405f5d22&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=b0e7999c9d0d472195e1c5dc6f4d4978 => generated 611 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:32.885996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:32.911390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:32.912136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:32.912484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:32.912820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:32.913836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:32.914310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:32.916135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:32.923339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:32.928539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 219/445] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:32.934408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:33.024826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351460d3-0c25-4849-a54d-276ad072f822 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.026528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:33.070889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c463c8-a29f-4d1c-a9d0-06c9cdd47e1c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.073400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b336a-f80c-456d-af09-7a139c6c1af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.073744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b336a-f80c-456d-af09-7a139c6c1af6 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.074029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b336a-f80c-456d-af09-7a139c6c1af6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.074308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b336a-f80c-456d-af09-7a139c6c1af6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.087901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2b336a-f80c-456d-af09-7a139c6c1af6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-7c2b336a-f80c-456d-af09-7a139c6c1af6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:33.093188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:33.094088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:33.094906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 227/446] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:33.102030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b336a-f80c-456d-af09-7a139c6c1af6 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:33.113650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c848bb46-2a08-47c2-b2dd-5fae647f0be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.114599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c848bb46-2a08-47c2-b2dd-5fae647f0be0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.115120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c848bb46-2a08-47c2-b2dd-5fae647f0be0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.117203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c848bb46-2a08-47c2-b2dd-5fae647f0be0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.197280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c848bb46-2a08-47c2-b2dd-5fae647f0be0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:33.198929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c848bb46-2a08-47c2-b2dd-5fae647f0be0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:33.199425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:33.222463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b336a-f80c-456d-af09-7a139c6c1af6 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.264361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.264978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.265258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.265532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.267724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:33.267724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:33.268773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:33.273221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:33.279301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 228/447] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:33.287383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:33.324734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 220/448] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:56:33.330248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c848bb46-2a08-47c2-b2dd-5fae647f0be0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:33.395343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0f348f-6d10-4f08-8893-1c03a909d383 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.428044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c848bb46-2a08-47c2-b2dd-5fae647f0be0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.428784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db45f19-5575-45ed-8291-671e0bf5aa0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.429399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db45f19-5575-45ed-8291-671e0bf5aa0c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.429681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db45f19-5575-45ed-8291-671e0bf5aa0c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.429955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db45f19-5575-45ed-8291-671e0bf5aa0c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.447210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:33.461702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d4aa8-eaf4-49e3-91a1-93ab38575360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.462321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d4aa8-eaf4-49e3-91a1-93ab38575360 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.462597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d4aa8-eaf4-49e3-91a1-93ab38575360 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.462867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d4aa8-eaf4-49e3-91a1-93ab38575360 None admin] PATH_INFO: `/v3/projects/6fa314f8793248fb8bbde6daa676e4e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.464002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861d4aa8-eaf4-49e3-91a1-93ab38575360 None admin] RBAC: Authorizing `identity:get_project(project_id=6fa314f8793248fb8bbde6daa676e4e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:33.465429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861d4aa8-eaf4-49e3-91a1-93ab38575360 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:33.467014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 221/449] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:33 2022] GET /identity/v3/projects/6fa314f8793248fb8bbde6daa676e4e7 => generated 302 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:33.473309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d4aa8-eaf4-49e3-91a1-93ab38575360 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:33.497769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db45f19-5575-45ed-8291-671e0bf5aa0c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:33.499367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db45f19-5575-45ed-8291-671e0bf5aa0c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:33.503661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:33.573535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d4aa8-eaf4-49e3-91a1-93ab38575360 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.595906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 229/450] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:56:33.601430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db45f19-5575-45ed-8291-671e0bf5aa0c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:33.606919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668150d5-7dde-4ede-bf31-39ebce3b3885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.607536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668150d5-7dde-4ede-bf31-39ebce3b3885 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.607847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668150d5-7dde-4ede-bf31-39ebce3b3885 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.608114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668150d5-7dde-4ede-bf31-39ebce3b3885 None admin] PATH_INFO: `/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.609209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668150d5-7dde-4ede-bf31-39ebce3b3885 None admin] RBAC: Authorizing `identity:get_group(group_id=81ecc4767f864a7e9c5d7b5e405f5d22)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:33.610610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668150d5-7dde-4ede-bf31-39ebce3b3885 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:33.612535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 222/451] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:33 2022] GET /identity/v3/groups/81ecc4767f864a7e9c5d7b5e405f5d22 => generated 226 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:33.617906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668150d5-7dde-4ede-bf31-39ebce3b3885 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:33.653615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:33.705518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db45f19-5575-45ed-8291-671e0bf5aa0c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.712392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:33.716115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668150d5-7dde-4ede-bf31-39ebce3b3885 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.739236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b9e299-182a-4a61-bfce-a9705754cff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.739886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b9e299-182a-4a61-bfce-a9705754cff3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.740164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b9e299-182a-4a61-bfce-a9705754cff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.740436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b9e299-182a-4a61-bfce-a9705754cff3 None admin] PATH_INFO: `/v3/projects/f973480c94d94371b4cef62cd36d8410` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.741533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b9e299-182a-4a61-bfce-a9705754cff3 None admin] RBAC: Authorizing `identity:get_project(project_id=f973480c94d94371b4cef62cd36d8410)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:33.742997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b9e299-182a-4a61-bfce-a9705754cff3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:33.744624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 230/452] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:56:33 2022] GET /identity/v3/projects/f973480c94d94371b4cef62cd36d8410 => generated 344 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:33.749946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b9e299-182a-4a61-bfce-a9705754cff3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:33.754861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.755464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.755783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.756059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.756791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] RBAC: Adding query filter params (group.id=81ecc4767f864a7e9c5d7b5e405f5d22, role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=6fa314f8793248fb8bbde6daa676e4e7) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:33.757103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:33.758571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:33.764259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 223/453] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:33 2022] GET /identity/v3/role_assignments?group.id=81ecc4767f864a7e9c5d7b5e405f5d22&scope.project.id=6fa314f8793248fb8bbde6daa676e4e7&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:33.859440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b9e299-182a-4a61-bfce-a9705754cff3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:33.877531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:33.897843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5039a553-3ca4-445a-b430-49ba62a2fc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:33.898493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5039a553-3ca4-445a-b430-49ba62a2fc4d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:33.898768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5039a553-3ca4-445a-b430-49ba62a2fc4d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:33.899037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5039a553-3ca4-445a-b430-49ba62a2fc4d None admin] PATH_INFO: `/v3/groups/bb6c07e9b1d7471bbe04d1115a58efa5` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:33.900241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5039a553-3ca4-445a-b430-49ba62a2fc4d None admin] RBAC: Authorizing `identity:get_group(group_id=bb6c07e9b1d7471bbe04d1115a58efa5)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:33.901751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5039a553-3ca4-445a-b430-49ba62a2fc4d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:33.903349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 231/454] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 15 13:56:33 2022] GET /identity/v3/groups/bb6c07e9b1d7471bbe04d1115a58efa5 => generated 229 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:33.910162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:34.010870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5a03aa-1522-4585-8f03-90bfef654472 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:34.045253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57b3859-2b33-4a94-8fb6-cab0807bcecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:34.045858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57b3859-2b33-4a94-8fb6-cab0807bcecb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:34.046138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57b3859-2b33-4a94-8fb6-cab0807bcecb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:34.046432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57b3859-2b33-4a94-8fb6-cab0807bcecb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:34.047155 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57b3859-2b33-4a94-8fb6-cab0807bcecb None admin] RBAC: Adding query filter params (group.id=bb6c07e9b1d7471bbe04d1115a58efa5, role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=f973480c94d94371b4cef62cd36d8410) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:34.047465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57b3859-2b33-4a94-8fb6-cab0807bcecb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:34.048925 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57b3859-2b33-4a94-8fb6-cab0807bcecb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:34.055488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 224/455] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 15 13:56:33 2022] GET /identity/v3/role_assignments?group.id=bb6c07e9b1d7471bbe04d1115a58efa5&scope.project.id=f973480c94d94371b4cef62cd36d8410&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:36.792233 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32882bd-d43f-46ae-9acc-47414e21c014 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:36.793607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32882bd-d43f-46ae-9acc-47414e21c014 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:36.794106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32882bd-d43f-46ae-9acc-47414e21c014 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:36.796544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 232/456] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:36.802416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd5faa-0d52-4121-b423-c19ca19c6553 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:36.803276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd5faa-0d52-4121-b423-c19ca19c6553 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:36.803761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd5faa-0d52-4121-b423-c19ca19c6553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:36.811369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1437305-8ca4-4934-80f5-9ee90afcf33d None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:36.811760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1437305-8ca4-4934-80f5-9ee90afcf33d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:36.812108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1437305-8ca4-4934-80f5-9ee90afcf33d None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:36.813958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 233/457] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:36.826154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a66fd28-ee73-4dc8-b247-135c3c983b8f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:36.826624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a66fd28-ee73-4dc8-b247-135c3c983b8f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:36.827202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a66fd28-ee73-4dc8-b247-135c3c983b8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:36.832523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-21cd5faa-0d52-4121-b423-c19ca19c6553 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:36.833961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-21cd5faa-0d52-4121-b423-c19ca19c6553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:36.861006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6a66fd28-ee73-4dc8-b247-135c3c983b8f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:36.862270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6a66fd28-ee73-4dc8-b247-135c3c983b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:36.879984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:36.964715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 225/458] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:36.992982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 234/459] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:37.279636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd5faa-0d52-4121-b423-c19ca19c6553 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:37.314787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd1a5f4-aed2-469d-a1b7-e0db48942d71 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:37.314902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd1a5f4-aed2-469d-a1b7-e0db48942d71 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:37.315162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd1a5f4-aed2-469d-a1b7-e0db48942d71 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:37.318721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 235/460] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:37.322742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a40de8-0a89-4e2c-a214-edfa417570c5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:37.323027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a40de8-0a89-4e2c-a214-edfa417570c5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:37.323302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a40de8-0a89-4e2c-a214-edfa417570c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:37.353165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a1a40de8-0a89-4e2c-a214-edfa417570c5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:37.354364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a1a40de8-0a89-4e2c-a214-edfa417570c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:37.432111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd5faa-0d52-4121-b423-c19ca19c6553 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:37.438073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 236/461] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:37.438930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a40de8-0a89-4e2c-a214-edfa417570c5 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:37.487601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:37.495116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:37.495862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:37.496142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:37.496416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:37.507252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:37.508816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:37.524510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:37.525433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:37.526288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:37.526288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:37.526288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:37.526288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 226/462] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:37.532744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:37.553442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:37.591622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a40de8-0a89-4e2c-a214-edfa417570c5 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:37.661040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:37.661679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:37.661956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:37.662230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:37.685825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:37.687292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:37.719358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3879e2bc-a561-46d6-8f44-e0ec470c7a01 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:37.722541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:37.723473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:37.724457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:37.724457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:37.724457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:37.732189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 237/463] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:37.748727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:37.764411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:37.765024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:37.765305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:37.765582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:37.766242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:37.766549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:37.771709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:37.781326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 227/464] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:37.786577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:37.875062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:37.894321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:37.900634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead59cc-f4ed-443a-8002-340954082b0c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:37.913891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8960b5-a6ea-4463-b59b-d5ae84f17a31 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:37.936714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:37.937740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:37.938022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:37.938444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:37.939120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:37.939520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:37.941280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:37.952329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ea385a-08e4-4da5-a372-1b520d58639f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:37.952942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea385a-08e4-4da5-a372-1b520d58639f None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:37.953226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea385a-08e4-4da5-a372-1b520d58639f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:37.953498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea385a-08e4-4da5-a372-1b520d58639f None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:37.953921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 238/465] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:37.954145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ea385a-08e4-4da5-a372-1b520d58639f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:37.955568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ea385a-08e4-4da5-a372-1b520d58639f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:37.959083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:37.979356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 228/466] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:56:37 2022] POST /identity/v3/users => generated 248 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:56:37.980478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a9fb67-c77a-4229-8902-0cd91d90e60a None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:37.980768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a9fb67-c77a-4229-8902-0cd91d90e60a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:37.981042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a9fb67-c77a-4229-8902-0cd91d90e60a None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:37.982417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 229/467] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:37.988597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a9fb67-c77a-4229-8902-0cd91d90e60a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:38.088504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009e4a45-8308-4579-8ae9-2f6854edc5d7 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:38.133289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a9fb67-c77a-4229-8902-0cd91d90e60a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:38.135191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25168cd-25d1-4797-927a-3610d0f3e524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:38.135998 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25168cd-25d1-4797-927a-3610d0f3e524 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:38.136381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25168cd-25d1-4797-927a-3610d0f3e524 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:38.137566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25168cd-25d1-4797-927a-3610d0f3e524 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:38.151559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25168cd-25d1-4797-927a-3610d0f3e524 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:38.154089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25168cd-25d1-4797-927a-3610d0f3e524 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:38.164835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-e25168cd-25d1-4797-927a-3610d0f3e524 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:38.172856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:38.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:38.175259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:38.175259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:38.175259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:38.175259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 239/468] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:38.176537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ac02c-f027-491d-991a-01c2c0b66c60 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:38.176537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ac02c-f027-491d-991a-01c2c0b66c60 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:38.176537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ac02c-f027-491d-991a-01c2c0b66c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:38.186591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b20233-35fa-4996-9b9c-56e97ba36154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:38.187202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b20233-35fa-4996-9b9c-56e97ba36154 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:38.187480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b20233-35fa-4996-9b9c-56e97ba36154 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:38.189453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b20233-35fa-4996-9b9c-56e97ba36154 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:38.189453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b20233-35fa-4996-9b9c-56e97ba36154 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:38.190568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b20233-35fa-4996-9b9c-56e97ba36154 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:38.197425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2b4ac02c-f027-491d-991a-01c2c0b66c60 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:38.199197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2b4ac02c-f027-491d-991a-01c2c0b66c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:38.203793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:38.221122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 230/469] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:56:37 2022] POST /identity/v3/users => generated 246 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:56:38.223021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b20233-35fa-4996-9b9c-56e97ba36154 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:38.315861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 240/470] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:38.353195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc602952-8324-496f-b695-282ac04cff34 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:38.353439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc602952-8324-496f-b695-282ac04cff34 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:38.353672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc602952-8324-496f-b695-282ac04cff34 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:38.359350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 241/471] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:38.361132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b18c95-4670-452e-8ee7-90dd676d6759 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:38.361417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b18c95-4670-452e-8ee7-90dd676d6759 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:38.361691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b18c95-4670-452e-8ee7-90dd676d6759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:38.382888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-19b18c95-4670-452e-8ee7-90dd676d6759 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:38.384517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-19b18c95-4670-452e-8ee7-90dd676d6759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:38.388243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b20233-35fa-4996-9b9c-56e97ba36154 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:38.427764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:38.428475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:38.428887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:38.429220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:38.430002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:38.430365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:38.431904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:38.446513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 231/472] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:38.452291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:38.501881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:38.547788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 242/473] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:38.574451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f10f6-3a8d-4d1d-bfa3-998af2c5768b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:38.611588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939d51ed-dbf3-431b-848f-8f8623444f3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:38.612830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d51ed-dbf3-431b-848f-8f8623444f3c None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:38.613164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d51ed-dbf3-431b-848f-8f8623444f3c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:38.613498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d51ed-dbf3-431b-848f-8f8623444f3c None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:38.614254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939d51ed-dbf3-431b-848f-8f8623444f3c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:38.615755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939d51ed-dbf3-431b-848f-8f8623444f3c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:38.630976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b18c95-4670-452e-8ee7-90dd676d6759 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:38.650087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 232/474] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:56:38 2022] POST /identity/v3/users => generated 248 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:56:38.715279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:38.755810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b18c95-4670-452e-8ee7-90dd676d6759 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:38.794635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:38.795377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:38.795751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:38.796133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:38.808098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:38.809584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:38.827482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:38.828508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:38.829358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:38.829358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:38.829358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:38.829358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 243/475] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:38.835162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939d51ed-dbf3-431b-848f-8f8623444f3c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:38.849983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:38.955171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:38.964753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939d51ed-dbf3-431b-848f-8f8623444f3c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:38.996448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c159b60f-afa2-41b7-9d07-997a6c333470 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:39.012191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:39.013244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:39.013531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:39.013808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:39.014764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:39.015133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:39.016670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:39.026571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:39.035730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 233/476] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:39.039878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:39.048778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:39.050773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:39.051455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:39.051920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:39.053216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:39.055080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:39.067419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.sql.core [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 15 13:56:39.067419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 15 13:56:39.067419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [parameters: {'id': 'ca0a70a301d54d0b8f2811fe55e4af59', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 15 13:56:39.067419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81655) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application result = self._query(query) Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application conn.query(q) Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application result.read() Sep 15 13:56:39.092664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return ref.to_dict() Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 15 13:56:39.093781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application next(self.gen) Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application yield resource Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application next(self.gen) Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self.session.rollback() Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self.force_reraise() Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise self.value Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application session.commit() Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 15 13:56:39.094929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._prepare_impl() Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self.session.flush() Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._flush(objects) Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application compat.raise_( Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application flush_context.execute() Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rec.execute(self) Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 15 13:56:39.096766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 15 13:56:39.099324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application result = self._query(query) Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application conn.query(q) Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application result.read() Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 15 13:56:39.102108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [parameters: {'id': 'ca0a70a301d54d0b8f2811fe55e4af59', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:39.104970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 15 13:56:39.107310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:39.109739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 244/477] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 15 13:56:38 2022] POST /identity/v3/roles => generated 144 bytes in 248 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:39.109739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:39.195115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1e661-e98e-4c18-b8c4-74aff23582b3 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:39.234503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb247e9-c5fe-4924-964e-8dc5180ec0f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:39.235213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb247e9-c5fe-4924-964e-8dc5180ec0f8 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:39.235458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb247e9-c5fe-4924-964e-8dc5180ec0f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:39.235770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb247e9-c5fe-4924-964e-8dc5180ec0f8 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:39.236432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb247e9-c5fe-4924-964e-8dc5180ec0f8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:39.238472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb247e9-c5fe-4924-964e-8dc5180ec0f8 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:39.268911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 234/478] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:56:39 2022] POST /identity/v3/users => generated 249 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:56:39.326003 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0af71f-8ea4-4974-ad42-ef7bf5058285 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:39.383723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2966b88-aade-4b34-879d-d90f860b4614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:39.384455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2966b88-aade-4b34-879d-d90f860b4614 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:39.384737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2966b88-aade-4b34-879d-d90f860b4614 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:39.385013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2966b88-aade-4b34-879d-d90f860b4614 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:39.407287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2966b88-aade-4b34-879d-d90f860b4614 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d2966b88-aade-4b34-879d-d90f860b4614 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:39.413853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:39.416455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:39.418389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 245/479] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:56:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:39.422697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb247e9-c5fe-4924-964e-8dc5180ec0f8 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:39.451942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:39.569664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb247e9-c5fe-4924-964e-8dc5180ec0f8 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:39.605967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bcfc70-2157-45fd-8ae4-1640135376a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:39.606579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bcfc70-2157-45fd-8ae4-1640135376a6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:39.606856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bcfc70-2157-45fd-8ae4-1640135376a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:39.607125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bcfc70-2157-45fd-8ae4-1640135376a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:39.607891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bcfc70-2157-45fd-8ae4-1640135376a6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:39.608193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bcfc70-2157-45fd-8ae4-1640135376a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:39.609547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bcfc70-2157-45fd-8ae4-1640135376a6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:39.620681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 235/480] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:40.820873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc33c14-f474-4d43-b86d-e384abb74caf None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:40.821865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc33c14-f474-4d43-b86d-e384abb74caf None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:40.822379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc33c14-f474-4d43-b86d-e384abb74caf None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:40.824242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 246/481] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:40.829261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe5d59-1fe1-4dfc-90be-31f5b496c830 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:40.830064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe5d59-1fe1-4dfc-90be-31f5b496c830 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:40.830541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe5d59-1fe1-4dfc-90be-31f5b496c830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:40.860673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9bbe5d59-1fe1-4dfc-90be-31f5b496c830 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:40.862737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9bbe5d59-1fe1-4dfc-90be-31f5b496c830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:40.953318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 236/482] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:41.117769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723bdf41-8534-482b-bf37-78cb9263522c None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:41.118374 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723bdf41-8534-482b-bf37-78cb9263522c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:41.118869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723bdf41-8534-482b-bf37-78cb9263522c None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:41.120442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 247/483] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:41.126022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f045fe2e-a408-4705-a45a-3b106db02d9b None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:41.126575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f045fe2e-a408-4705-a45a-3b106db02d9b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:41.127060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f045fe2e-a408-4705-a45a-3b106db02d9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:41.140464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:41.156795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f045fe2e-a408-4705-a45a-3b106db02d9b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:41.158353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f045fe2e-a408-4705-a45a-3b106db02d9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:41.252168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 237/484] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:41.316312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723bdf41-8534-482b-bf37-78cb9263522c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:41.381491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:41.425779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5761aa6-f8b2-4d5d-9f64-118df0622cc9 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:41.426724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5761aa6-f8b2-4d5d-9f64-118df0622cc9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:41.427197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5761aa6-f8b2-4d5d-9f64-118df0622cc9 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:41.433187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 238/485] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:41 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:41.438743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18970a0-6034-4413-9647-39340065b655 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:41.439372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18970a0-6034-4413-9647-39340065b655 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:41.439993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18970a0-6034-4413-9647-39340065b655 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:41.449861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723bdf41-8534-482b-bf37-78cb9263522c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:41.476817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b18970a0-6034-4413-9647-39340065b655 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:41.478530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b18970a0-6034-4413-9647-39340065b655 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:41.490840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05509e8-551d-4e89-afb7-f61c05642106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:41.491548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05509e8-551d-4e89-afb7-f61c05642106 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:41.491853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05509e8-551d-4e89-afb7-f61c05642106 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:41.492124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05509e8-551d-4e89-afb7-f61c05642106 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:41.505418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05509e8-551d-4e89-afb7-f61c05642106 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-b05509e8-551d-4e89-afb7-f61c05642106 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:41.510382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:41.511245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:41.512082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 248/486] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:41.517221 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05509e8-551d-4e89-afb7-f61c05642106 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:41.622435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 239/487] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:41.634963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18970a0-6034-4413-9647-39340065b655 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:41.698890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05509e8-551d-4e89-afb7-f61c05642106 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:41.716008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:41.748311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:41.764615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:41.765227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:41.765508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:41.765779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:41.766444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:41.766747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:41.768291 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:41.783910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 249/488] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:41.787828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:41.817381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18970a0-6034-4413-9647-39340065b655 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:41.880501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98618e91-fff6-4819-aa26-c2357ce7f8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:41.881150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98618e91-fff6-4819-aa26-c2357ce7f8dd None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:41.881394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98618e91-fff6-4819-aa26-c2357ce7f8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:41.881670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98618e91-fff6-4819-aa26-c2357ce7f8dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:41.904570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98618e91-fff6-4819-aa26-c2357ce7f8dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-98618e91-fff6-4819-aa26-c2357ce7f8dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:41.910217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:41.911104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:41.912299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:41.913705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 240/489] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:41.920127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98618e91-fff6-4819-aa26-c2357ce7f8dd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:41.955092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979f4ec-a94a-4d59-9468-690a463d1539 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.007881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.009103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.009558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.010000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.025211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.027759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:42.046932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:42.047897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:42.048751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:42.048751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:42.048751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:42.050803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 250/490] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:42.052149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.065868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98618e91-fff6-4819-aa26-c2357ce7f8dd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.076525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:42.094227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:42.106295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.107093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.107537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.107994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.108798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:42.109241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.110763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.123608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 241/491] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:42.124643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.164515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801f279e-05ed-4d78-8755-c1cfa17edcaa None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.199470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5360ea3a-7eeb-4d17-bc17-c86a57659a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.200133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5360ea3a-7eeb-4d17-bc17-c86a57659a80 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.200414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5360ea3a-7eeb-4d17-bc17-c86a57659a80 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.200686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5360ea3a-7eeb-4d17-bc17-c86a57659a80 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.211845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5360ea3a-7eeb-4d17-bc17-c86a57659a80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-5360ea3a-7eeb-4d17-bc17-c86a57659a80 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:42.217009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:42.217911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:42.218706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 251/492] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:42.219392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ec997-ac09-40b9-88b1-5a8b5329bb25 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.219460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ec997-ac09-40b9-88b1-5a8b5329bb25 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.225007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ec997-ac09-40b9-88b1-5a8b5329bb25 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.225007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 252/493] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:42.225007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46ec997-ac09-40b9-88b1-5a8b5329bb25 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.238771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0f213-da19-4fa9-9519-fcb281853f80 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.276063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.276667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.276950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.277227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.277868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:42.278178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.279561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.292052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 242/494] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:42.292849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.316927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:42.341807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46ec997-ac09-40b9-88b1-5a8b5329bb25 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.352069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:42.377562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd1e311-42b4-4da1-bfd7-04947eeccf6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.379049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd1e311-42b4-4da1-bfd7-04947eeccf6b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.379327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd1e311-42b4-4da1-bfd7-04947eeccf6b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.379953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd1e311-42b4-4da1-bfd7-04947eeccf6b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.390335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd1e311-42b4-4da1-bfd7-04947eeccf6b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.391799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd1e311-42b4-4da1-bfd7-04947eeccf6b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3fd1e311-42b4-4da1-bfd7-04947eeccf6b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:42.406436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:42.407356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:42.408636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:42.408636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:42.408636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:42.408636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 253/495] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:42.409435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e8a410-8ab8-49e1-b515-cffa279b8686 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.409973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e8a410-8ab8-49e1-b515-cffa279b8686 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.409973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e8a410-8ab8-49e1-b515-cffa279b8686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.428699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-55e8a410-8ab8-49e1-b515-cffa279b8686 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:42.429182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de66288-cff3-4166-997d-afb81e83cd92 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.429807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-55e8a410-8ab8-49e1-b515-cffa279b8686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:42.468044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.468643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.468922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.469192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.469852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:42.470150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.472045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.480960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 243/496] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:42.481571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.508111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 254/497] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:42.508989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e8a410-8ab8-49e1-b515-cffa279b8686 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.590028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:42.608173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597d20ee-5fcb-4fe6-bb30-88ee1dc1752f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.618094 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:42.635478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e8a410-8ab8-49e1-b515-cffa279b8686 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.646887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.647506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.647886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.648164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] PATH_INFO: `/v3/users/glance` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.655747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.657176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:42.669858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:42.670772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 15 13:56:42.672255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:42.672764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c52df5-319e-4fb6-8a5c-388e4ae42cc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.672764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c52df5-319e-4fb6-8a5c-388e4ae42cc6 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.672764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c52df5-319e-4fb6-8a5c-388e4ae42cc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.672964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 244/498] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/users/glance => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:42.672964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.675581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c52df5-319e-4fb6-8a5c-388e4ae42cc6 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.676424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c52df5-319e-4fb6-8a5c-388e4ae42cc6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:42.676734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c52df5-319e-4fb6-8a5c-388e4ae42cc6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.678126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c52df5-319e-4fb6-8a5c-388e4ae42cc6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.693842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 255/499] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:42.693842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3cdeaf-f24d-4ac4-a6c5-d62756601593 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.693842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3cdeaf-f24d-4ac4-a6c5-d62756601593 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.693842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3cdeaf-f24d-4ac4-a6c5-d62756601593 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.693842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 256/500] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:42.695201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3cdeaf-f24d-4ac4-a6c5-d62756601593 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.789358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75514d-b3a0-4ccd-a595-4a6b0184e6ff None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.805511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3cdeaf-f24d-4ac4-a6c5-d62756601593 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.824141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.824762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.825069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.825347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.825910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:42.835712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.837179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.840433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ef0075-094b-4eb9-8163-f4f7e1c2b967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.841028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef0075-094b-4eb9-8163-f4f7e1c2b967 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.841302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef0075-094b-4eb9-8163-f4f7e1c2b967 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.841570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef0075-094b-4eb9-8163-f4f7e1c2b967 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.842284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ef0075-094b-4eb9-8163-f4f7e1c2b967 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:42.842584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ef0075-094b-4eb9-8163-f4f7e1c2b967 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:42.844010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ef0075-094b-4eb9-8163-f4f7e1c2b967 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:42.851184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:42.852296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:42.853187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:42.853187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:42.853187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:42.853187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 245/501] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:42.853401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.874801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:42.884440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 257/502] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:42.885674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563f0dd-66f4-41ff-9e6a-6fb97a631aeb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.885956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563f0dd-66f4-41ff-9e6a-6fb97a631aeb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.886225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563f0dd-66f4-41ff-9e6a-6fb97a631aeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.903916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c563f0dd-66f4-41ff-9e6a-6fb97a631aeb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:42.905110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c563f0dd-66f4-41ff-9e6a-6fb97a631aeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:42.952671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbd9e90-70d8-42c8-8449-1c531a1c6a6a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:42.972958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 258/503] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:42.973796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563f0dd-66f4-41ff-9e6a-6fb97a631aeb None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:42.990210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:42.990830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:42.991108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:42.992068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:42.998971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.000505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:43.012545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:43.013672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 246/504] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:43.014590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.091614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563f0dd-66f4-41ff-9e6a-6fb97a631aeb None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.101543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.102505 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.123460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed35aae-de08-4e3d-9278-5e17e161c716 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.129002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047c8ed9-f2fa-4281-a347-c36645f4af3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.129726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047c8ed9-f2fa-4281-a347-c36645f4af3e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.130008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047c8ed9-f2fa-4281-a347-c36645f4af3e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.130279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047c8ed9-f2fa-4281-a347-c36645f4af3e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.140943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047c8ed9-f2fa-4281-a347-c36645f4af3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-047c8ed9-f2fa-4281-a347-c36645f4af3e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:43.146420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:43.147308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:43.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 259/505] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:43.149402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047c8ed9-f2fa-4281-a347-c36645f4af3e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.163581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.164512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.164942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.165218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.165855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:43.166164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.167515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.179234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 247/506] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:43.180053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.262678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047c8ed9-f2fa-4281-a347-c36645f4af3e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.288710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16540d6f-b480-4255-8158-432c2a8a4be9 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.295422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135b4496-c52e-4c7c-bdf8-46a6481432d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.296049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135b4496-c52e-4c7c-bdf8-46a6481432d1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.296326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135b4496-c52e-4c7c-bdf8-46a6481432d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.296595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135b4496-c52e-4c7c-bdf8-46a6481432d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.325781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.326475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.326793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.327072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.327834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:43.328143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.329579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.332817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.354992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.366906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 248/507] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:43.367055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135b4496-c52e-4c7c-bdf8-46a6481432d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.367740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.368419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135b4496-c52e-4c7c-bdf8-46a6481432d1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.467500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b206ec40-a37c-4dc4-9786-373d6a694248 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.467809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 260/508] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:43.468377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135b4496-c52e-4c7c-bdf8-46a6481432d1 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.503667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.504293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.504570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.504842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.505510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:43.505815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.507209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.514881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 249/509] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:43.515586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.542281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.574708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135b4496-c52e-4c7c-bdf8-46a6481432d1 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.575538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.608611 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.609327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.609609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.609883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.616197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361df58e-b8c1-4708-aa09-930da0bf6599 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.617360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.618802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:43.631589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:43.632664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 261/510] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/users/cinder => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:43.633538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.654578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.655200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.655481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.655779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.665845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.667290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:43.682318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:43.683382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:43.684544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:43.684544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:43.684544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:43.684544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 250/511] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:43.684544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.736380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b026e558-e9ae-4923-9e00-3b335692543f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.761993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.771216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648acb00-70da-4d4f-80ab-043a7dc54df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.771788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648acb00-70da-4d4f-80ab-043a7dc54df4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.772417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648acb00-70da-4d4f-80ab-043a7dc54df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.772417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648acb00-70da-4d4f-80ab-043a7dc54df4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.783605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd39e68-4f10-4d1a-ab72-9da1432bd0f9 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.796289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.817987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.818605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.818884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.819158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.829294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.831273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.840315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648acb00-70da-4d4f-80ab-043a7dc54df4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.841761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648acb00-70da-4d4f-80ab-043a7dc54df4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:43.845028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:43.846056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:43.847547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:43.847547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:43.847547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:43.847547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 251/512] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:43.847882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.942434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 262/513] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:43.943283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648acb00-70da-4d4f-80ab-043a7dc54df4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:43.946353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86174322-ce3a-4d4d-a36c-1d3699c316ef None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:43.970356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:43.979518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:43.980175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:43.980453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:43.980741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:43.991217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:43.992661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:44.006997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:44.007978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:44.008833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:44.008833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:44.008833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:44.008833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 252/514] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:44.008833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.027288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.045855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648acb00-70da-4d4f-80ab-043a7dc54df4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.079376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.080006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.080287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.080559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.081539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:44.081854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.083328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.100563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 263/515] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 15 13:56:43 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:44.104510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.108580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213611bd-023e-4353-b575-41041c2aa507 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.141947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.142547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.142831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.143103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.143920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:44.144225 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.145604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.157397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 253/516] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:44.158216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.188802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.204136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa327ca-a183-4b55-87bc-873568e60b7f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.235566 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.238838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.239448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.239754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.240030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.240673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:44.241030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.242453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.253901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 264/517] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:44.255010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.264361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6737f164-f920-40b1-ab2e-6df884485250 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.300396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.301265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.301590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.301910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.312353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.314142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:44.330133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:44.331228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:44.332313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:44.332313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:44.332313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:44.332313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 254/518] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:44.332531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.361567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67bcd3b-88f4-45f3-be31-f45afe206719 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.395345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.395984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.396269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.396547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.397189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:44.397496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.398870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.410939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 265/519] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:44.411730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.419787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.432732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e466d4ed-c1e4-4fb9-bead-6005edd22119 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.458328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.466551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e7e49a-ad9a-45cf-a134-010ad934757d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.467153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e7e49a-ad9a-45cf-a134-010ad934757d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.467430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e7e49a-ad9a-45cf-a134-010ad934757d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.467735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e7e49a-ad9a-45cf-a134-010ad934757d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.510839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc2a196-d680-4e10-86eb-2129746a1e1a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.534365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e7e49a-ad9a-45cf-a134-010ad934757d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.535837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e7e49a-ad9a-45cf-a134-010ad934757d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.545135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d3a99f-a5ed-472c-beed-119270719e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.545738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d3a99f-a5ed-472c-beed-119270719e3a None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.546023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d3a99f-a5ed-472c-beed-119270719e3a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.546298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d3a99f-a5ed-472c-beed-119270719e3a None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.546938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d3a99f-a5ed-472c-beed-119270719e3a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.548379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d3a99f-a5ed-472c-beed-119270719e3a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.570333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 266/520] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:56:44 2022] POST /identity/v3/users => generated 247 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:56:44.571060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d3a99f-a5ed-472c-beed-119270719e3a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.644853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 255/521] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:44.645603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e7e49a-ad9a-45cf-a134-010ad934757d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.675145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.677626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.730535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d3a99f-a5ed-472c-beed-119270719e3a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.762811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e7e49a-ad9a-45cf-a134-010ad934757d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.770963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.771600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.771927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.772482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.780613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.782834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:44.799654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:44.800609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:44.802000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 267/522] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:44.802741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.807308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.807967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.808249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.808520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.809165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:44.809465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.810819 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.827899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 256/523] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:44.828943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.912436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b6abe6-f43b-4089-b1c2-b5295391b56f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.920942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.936251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:44.936417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f2d4c-9911-4747-a5ca-4bc7adf2a5ff None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:44.949969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.950597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.950868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.951145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.959958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.961412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:44.972974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:44.973462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:44.974303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 15 13:56:44.975832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:44.976347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:44.976347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:44.976347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:44.976598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 268/524] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/users/neutron => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:44.976598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:44.990671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:44.992161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:45.007128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:45.008066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:45.008912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:45.008912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:45.008912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:45.008912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 257/525] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:45.009140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.083798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be961830-cb56-4354-8044-e43305d4d1b3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.114355 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c7c18-1bcb-45f4-819a-7d2b6e2a2b1d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.120970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.121987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.122265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.122536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.123575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:45.123913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.125654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.144172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 269/526] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:45.145184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.153137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.153778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.154061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.154338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.161551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.163003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.165573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:45.178354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:45.179228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:45.180119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 258/527] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:45.181257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.198109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:45.256459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4cee9-03ba-4391-8b3c-87390fe60901 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.289274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a968f217-8dde-4590-aad6-ac253a1ea1e0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.291928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.292540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.292824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.293098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.293827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:45.294134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.295572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.322010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 270/528] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:45.322010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.339476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.340149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.340495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.340842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.341555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:45.341925 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.343363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.360624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 259/529] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:45.360624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.420379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:45.431835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:45.435537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5575e470-4a2e-487d-a1e8-fc29d85d592c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.464161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e608be-9055-4db0-bf6b-314bb9059c8c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.472819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.473432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.473724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.473999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.474728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=54db2e6805fd40f79c03eb3b2319f2b5) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:45.475040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.476517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.481990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 271/530] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/role_assignments?user.id=54db2e6805fd40f79c03eb3b2319f2b5&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:45.482789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.519897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.520918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.521394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.522204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.523658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:45.524482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.526402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.554218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 260/531] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:45.554218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.635446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6aabfc-a64b-453c-8cc2-73f61bc8be90 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.683123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aee170-9f68-431d-a4b9-d0116b829e8f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.683312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddbd2c1-38ab-4183-90bf-fe92f030075c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.683833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbd2c1-38ab-4183-90bf-fe92f030075c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.684120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbd2c1-38ab-4183-90bf-fe92f030075c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.684396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbd2c1-38ab-4183-90bf-fe92f030075c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.708793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:45.727294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:45.727840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.728580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.728759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.729110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.740799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.743167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:45.758381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:45.759281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:45.760178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:45.760178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:45.760178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:45.760178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:45.760178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:45.760178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:45.760178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:45.760178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:45.764564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 261/532] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:45.765414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.781692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddbd2c1-38ab-4183-90bf-fe92f030075c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.784609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddbd2c1-38ab-4183-90bf-fe92f030075c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.897472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705324e5-e9f6-4bed-a7eb-23de345f9e09 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:45.897828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 272/533] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:45.898279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddbd2c1-38ab-4183-90bf-fe92f030075c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.945902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:45.947315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:45.948272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:45.948757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:45.949973 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=ce6744790db74f0aaffc6eb269bf6737) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:45.950740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:45.953181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:45.961152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 262/534] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/role_assignments?user.id=ce6744790db74f0aaffc6eb269bf6737&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:45.962350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:45.984519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:46.029324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddbd2c1-38ab-4183-90bf-fe92f030075c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:46.069259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:46.070386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:46.070869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:46.071390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:46.072378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:46.072886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:46.074564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:46.077875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:46.091645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 273/535] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:46.098158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:46.184253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22420f0b-a48b-4400-8939-1ca12c7a1fc3 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:46.222895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:46.223474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:46.223783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:46.224058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:46.235200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:46.236856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:46.245338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78493b2b-1133-46ad-b27b-caf86310ea6b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:46.251606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:46.253085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:46.253943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:46.253943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:46.253943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:46.261260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 263/536] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:46.266540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:46.287712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:46.295057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:46.295984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:46.296470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:46.296940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:46.297874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=f54979e322484a42bb57d1b3a619652c) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:46.298473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:46.300211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:46.319807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 274/537] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:46 2022] GET /identity/v3/role_assignments?user.id=f54979e322484a42bb57d1b3a619652c&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:46.449621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:46.457685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfeea8f5-3184-48b5-8a1b-ad484ceafa2a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:46.508753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:46.509377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:46.509660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:46.509934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:46.510592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:46.510898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:46.512332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:46.537843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 264/538] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:46.543358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:46.668038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53085512-b38d-4cdb-9c71-b0e535944d08 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:46.709065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898f6b40-577b-4543-ba4b-18188d2188a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:46.709732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f6b40-577b-4543-ba4b-18188d2188a5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:46.710017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f6b40-577b-4543-ba4b-18188d2188a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:46.710295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898f6b40-577b-4543-ba4b-18188d2188a5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:46.718356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898f6b40-577b-4543-ba4b-18188d2188a5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:46.719889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898f6b40-577b-4543-ba4b-18188d2188a5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-898f6b40-577b-4543-ba4b-18188d2188a5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:46.733044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:46.733938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:46.734781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 275/539] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:46.741287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:46.824534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57306ea8-efe0-4994-94e7-68c7afc0c6b7 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:46.824857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57306ea8-efe0-4994-94e7-68c7afc0c6b7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:46.825110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57306ea8-efe0-4994-94e7-68c7afc0c6b7 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:46.826732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 276/540] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:46.832699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38730f-6092-4eb7-9b0b-2e840000fb39 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:46.832986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38730f-6092-4eb7-9b0b-2e840000fb39 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:46.833266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38730f-6092-4eb7-9b0b-2e840000fb39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:46.867566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ee38730f-6092-4eb7-9b0b-2e840000fb39 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:46.868805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ee38730f-6092-4eb7-9b0b-2e840000fb39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:46.874531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2589676-7618-41dd-a98e-60389865d87b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:46.919051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:46.920092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:46.921141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:46.922334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:46.924223 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:46.925718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:46.929691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:46.951222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 265/541] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:46.957185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:46.972151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:46.992062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:47.017240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 277/542] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 15 13:56:47.102348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1983108f-1123-408e-ba07-97d16af18b03 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:47.138575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:47.139204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:47.139488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:47.139786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:47.140548 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=922012eacf244f9480c180c2eb9eaf28) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:47.140858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:47.142326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:47.149620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 266/543] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:46 2022] GET /identity/v3/role_assignments?user.id=922012eacf244f9480c180c2eb9eaf28&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:47.332082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee38730f-6092-4eb7-9b0b-2e840000fb39 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:47.462941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee38730f-6092-4eb7-9b0b-2e840000fb39 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:47.500100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9abe00a-f326-4c32-a7a4-f898235f0afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:47.500738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9abe00a-f326-4c32-a7a4-f898235f0afb None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:47.501021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9abe00a-f326-4c32-a7a4-f898235f0afb None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:47.501297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9abe00a-f326-4c32-a7a4-f898235f0afb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:47.512162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9abe00a-f326-4c32-a7a4-f898235f0afb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a9abe00a-f326-4c32-a7a4-f898235f0afb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:47.517472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:47.518948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:47.519950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 278/544] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:47.524605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:47.644138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:47.670401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d765ad-b3ec-47a1-a3e5-e279d3326675 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:47.714009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:47.714589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:47.714875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:47.715153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:47.715866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:47.716174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:47.717542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:47.726936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 267/545] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:47.732205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9abe00a-f326-4c32-a7a4-f898235f0afb None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:47.788878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:47.868801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9abe00a-f326-4c32-a7a4-f898235f0afb None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:47.916299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:47.916865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:47.917156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:47.917442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:47.933036 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:47.934538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:47.956448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:47.957407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:47.958241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:47.958241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:47.958241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:47.958241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 279/546] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:47.963202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:48.076022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a784b1a1-106a-4966-a816-cca152d78b90 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:48.119258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1426966-fba1-423c-b7ab-0d824fb4b44c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:48.119903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1426966-fba1-423c-b7ab-0d824fb4b44c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:48.120146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1426966-fba1-423c-b7ab-0d824fb4b44c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:48.120423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1426966-fba1-423c-b7ab-0d824fb4b44c None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:48.121064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1426966-fba1-423c-b7ab-0d824fb4b44c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:48.121364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1426966-fba1-423c-b7ab-0d824fb4b44c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:48.122716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1426966-fba1-423c-b7ab-0d824fb4b44c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:48.135778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 268/547] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:48.141231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:48.271266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:48.285622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d652308-dde9-43da-ab1a-c73931be0d01 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:48.292926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45270166-e735-4654-b77e-a4a5d3a9d3ee None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:48.293210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45270166-e735-4654-b77e-a4a5d3a9d3ee None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:48.293481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45270166-e735-4654-b77e-a4a5d3a9d3ee None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:48.295079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 269/548] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:48.307352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc49a5ae-9679-4cec-b638-8597255fe0fd None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:48.307770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc49a5ae-9679-4cec-b638-8597255fe0fd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:48.308044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc49a5ae-9679-4cec-b638-8597255fe0fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:48.321081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:48.336420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-dc49a5ae-9679-4cec-b638-8597255fe0fd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:48.337970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dc49a5ae-9679-4cec-b638-8597255fe0fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:48.363361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:48.365715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:48.366846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:48.367915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:48.381928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:48.384275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:48.417558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:48.419260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:56:48.420305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:48.428584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 280/549] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:56:48 2022] GET /identity/v3/users/swift => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:48.433705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:48.463298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 270/550] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:48.591647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd3d122-81cb-494e-904b-ffc8ce2e52c8 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:48.628512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1227d486-5bd0-4a7d-995d-4456e1afb88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:48.629553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1227d486-5bd0-4a7d-995d-4456e1afb88f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:48.630079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1227d486-5bd0-4a7d-995d-4456e1afb88f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:48.630542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1227d486-5bd0-4a7d-995d-4456e1afb88f None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:48.631504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1227d486-5bd0-4a7d-995d-4456e1afb88f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:48.632047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1227d486-5bd0-4a7d-995d-4456e1afb88f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:48.633786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1227d486-5bd0-4a7d-995d-4456e1afb88f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:48.654037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 281/551] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 15 13:56:48 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:48.661177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc49a5ae-9679-4cec-b638-8597255fe0fd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:48.795314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc49a5ae-9679-4cec-b638-8597255fe0fd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:48.827152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:48.832902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16469449-303c-4576-a769-8e6eff0393ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:48.833671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16469449-303c-4576-a769-8e6eff0393ba None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:48.834074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16469449-303c-4576-a769-8e6eff0393ba None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:48.834686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16469449-303c-4576-a769-8e6eff0393ba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:48.842418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d3650e-62e5-40bf-9aad-282786cb5209 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:48.842743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d3650e-62e5-40bf-9aad-282786cb5209 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:48.843135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d3650e-62e5-40bf-9aad-282786cb5209 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:48.844855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 282/552] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:48.861024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c099c168-6e00-4394-baf6-97295ed0a756 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:48.861300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c099c168-6e00-4394-baf6-97295ed0a756 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:48.861580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c099c168-6e00-4394-baf6-97295ed0a756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:48.903320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c099c168-6e00-4394-baf6-97295ed0a756 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:48.904602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c099c168-6e00-4394-baf6-97295ed0a756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:48.906753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:48.914814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16469449-303c-4576-a769-8e6eff0393ba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:48.916885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16469449-303c-4576-a769-8e6eff0393ba None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:48.995653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 283/553] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:48.996551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c099c168-6e00-4394-baf6-97295ed0a756 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.035288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 271/554] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:49.044995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16469449-303c-4576-a769-8e6eff0393ba None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.092790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:49.100159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c099c168-6e00-4394-baf6-97295ed0a756 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.138211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbc4c70-9c0c-40d7-b4d2-8dd55075547b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.138846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc4c70-9c0c-40d7-b4d2-8dd55075547b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.139230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc4c70-9c0c-40d7-b4d2-8dd55075547b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.139509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc4c70-9c0c-40d7-b4d2-8dd55075547b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.146958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:49.153008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbc4c70-9c0c-40d7-b4d2-8dd55075547b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-cdbc4c70-9c0c-40d7-b4d2-8dd55075547b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:49.158190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:49.160740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:49.162227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 284/555] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:49.167223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bf420d-01fd-4f19-9eb9-d88f0b3dbc40 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.167426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bf420d-01fd-4f19-9eb9-d88f0b3dbc40 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.169461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bf420d-01fd-4f19-9eb9-d88f0b3dbc40 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.169711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 285/556] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:49 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:49.170451 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bf420d-01fd-4f19-9eb9-d88f0b3dbc40 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.180308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16469449-303c-4576-a769-8e6eff0393ba None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.218236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d442cea8-3464-4dd0-8c63-5463f0ec2670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.219012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d442cea8-3464-4dd0-8c63-5463f0ec2670 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.219293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d442cea8-3464-4dd0-8c63-5463f0ec2670 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.219562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d442cea8-3464-4dd0-8c63-5463f0ec2670 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.231798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d442cea8-3464-4dd0-8c63-5463f0ec2670 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:49.233417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d442cea8-3464-4dd0-8c63-5463f0ec2670 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-d442cea8-3464-4dd0-8c63-5463f0ec2670 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:49.249725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:49.251877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:49.254514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:49.254514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:49.254514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:49.254514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 272/557] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:49.254514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91060c2-9430-43f0-9639-178ff3bd41e0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.254514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91060c2-9430-43f0-9639-178ff3bd41e0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.254514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91060c2-9430-43f0-9639-178ff3bd41e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.273698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c91060c2-9430-43f0-9639-178ff3bd41e0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:49.275053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c91060c2-9430-43f0-9639-178ff3bd41e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:49.282965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bf420d-01fd-4f19-9eb9-d88f0b3dbc40 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.319540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.320206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.320464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.320743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.321409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:49.321710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:49.323089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:49.337190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 286/558] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:49.338113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.361278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 273/559] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:49.362640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91060c2-9430-43f0-9639-178ff3bd41e0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.380142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:49.399515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:49.451562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea55856-754f-40b4-a1f1-dc339cacd220 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.478721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91060c2-9430-43f0-9639-178ff3bd41e0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.498181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.498800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.499084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.499360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.500038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:49.500339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:49.502388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:49.515367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.516155 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.516582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.516946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.519121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 287/560] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:49.519988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.536205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:49.537792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:49.553837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:49.554931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:49.555818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:49.555818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:49.555818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:49.556114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 274/561] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:49.557174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.646877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:49.652905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0377107-8b55-43ca-9a54-6e4a22f12405 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.676264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:49.694509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0b1f3-f579-433f-ba21-a0f97ee67bd4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.695086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d6c808-29d7-43d4-8e9e-069b3d474eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.695755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d6c808-29d7-43d4-8e9e-069b3d474eaf None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.696047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d6c808-29d7-43d4-8e9e-069b3d474eaf None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.696303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d6c808-29d7-43d4-8e9e-069b3d474eaf None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.707712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d6c808-29d7-43d4-8e9e-069b3d474eaf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-23d6c808-29d7-43d4-8e9e-069b3d474eaf None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:49.712617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:49.713520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 288/562] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:49.714362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d6c808-29d7-43d4-8e9e-069b3d474eaf None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.732703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.733330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.733607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.733879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.741398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:49.742830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:49.754580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:49.755456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 275/563] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:49.756727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.818055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d6c808-29d7-43d4-8e9e-069b3d474eaf None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.852360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b948616-a622-4193-86ff-4d39ff551c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.852962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b948616-a622-4193-86ff-4d39ff551c05 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.853236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b948616-a622-4193-86ff-4d39ff551c05 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.853508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b948616-a622-4193-86ff-4d39ff551c05 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.854145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b948616-a622-4193-86ff-4d39ff551c05 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:49.854444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b948616-a622-4193-86ff-4d39ff551c05 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:49.855259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dabd5eb-fa2e-4ca6-8753-3b775589079f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:49.855843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b948616-a622-4193-86ff-4d39ff551c05 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:49.866943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 289/564] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:49.868154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ef95-9a19-4e84-9846-8638f079b189 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.868445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ef95-9a19-4e84-9846-8638f079b189 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.868719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ef95-9a19-4e84-9846-8638f079b189 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.870185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 290/565] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:49.870896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4313ef95-9a19-4e84-9846-8638f079b189 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.883764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:49.892402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876cd6f2-6616-4add-8e0b-95101b655912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:49.893116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cd6f2-6616-4add-8e0b-95101b655912 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:49.894065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cd6f2-6616-4add-8e0b-95101b655912 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:49.894502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cd6f2-6616-4add-8e0b-95101b655912 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:49.905115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876cd6f2-6616-4add-8e0b-95101b655912 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-876cd6f2-6616-4add-8e0b-95101b655912 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:49.910350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:49.911458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 276/566] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:49.912466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876cd6f2-6616-4add-8e0b-95101b655912 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:49.914504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:49.974671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4313ef95-9a19-4e84-9846-8638f079b189 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.007708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.008324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.008604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.008872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.009523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:50.009819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.011179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.016625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876cd6f2-6616-4add-8e0b-95101b655912 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.019091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 291/567] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:50.019909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.050203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c37d6f-e5ad-4c08-bda9-6bb749d800bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.050798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c37d6f-e5ad-4c08-bda9-6bb749d800bb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.051329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c37d6f-e5ad-4c08-bda9-6bb749d800bb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.051603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c37d6f-e5ad-4c08-bda9-6bb749d800bb None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.052322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c37d6f-e5ad-4c08-bda9-6bb749d800bb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:50.052629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c37d6f-e5ad-4c08-bda9-6bb749d800bb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.054068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c37d6f-e5ad-4c08-bda9-6bb749d800bb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.066539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 277/568] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:50.067704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faf9fc0-111e-48ca-afbf-65db1b58ce00 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.067993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faf9fc0-111e-48ca-afbf-65db1b58ce00 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.068269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faf9fc0-111e-48ca-afbf-65db1b58ce00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.086163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4faf9fc0-111e-48ca-afbf-65db1b58ce00 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:50.087365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4faf9fc0-111e-48ca-afbf-65db1b58ce00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:50.118439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ac91bf-25a9-49de-875b-ebd88d413d4a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.119972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:50.130196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:50.153483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.154111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.154396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.154664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.154890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 278/569] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:50.155761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4faf9fc0-111e-48ca-afbf-65db1b58ce00 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.162555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.164031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:50.175151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:50.176386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:50.177357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 292/570] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/users/glance => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:50.177956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.265562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4faf9fc0-111e-48ca-afbf-65db1b58ce00 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.279277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c19a218-4e3c-4e02-af29-2729227da2c5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.307230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.307866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.308159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.308417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.309086 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:50.309800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.311162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.317324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.317958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.318238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.318513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.324547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 279/571] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:50.325213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.329826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.332158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:50.347336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:50.348331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:50.349301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:50.349301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:50.349301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:50.353740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 293/572] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:50.354598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.369990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:50.371753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:50.436666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402ed13b-1a21-493c-968a-a3ef696623ac None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.463245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a545d-b743-4d54-a810-7bd2fe633862 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.471747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.472345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.472623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.472894 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.473616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=30fc2a26b4fc4d24b984fa8097752055) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:50.473921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.475345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.480567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 280/573] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/role_assignments?user.id=30fc2a26b4fc4d24b984fa8097752055&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:50.481472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.497380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.498010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.498286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.498558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.499311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:50.499615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.501146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.524117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 294/574] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:50.525428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.603013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0711482-0093-4da1-8b92-e53622eb6730 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.604859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:50.619428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:50.641963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.642611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.642904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.643206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.653637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bf7a44-e6b2-4c3b-b0a6-605b19a270bf None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.654896 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.656594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:50.671802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:50.672985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:50.674092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:50.674092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:50.674092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:50.675532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 281/575] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:50.676775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.695825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.696368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.696647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.696918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.697559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:50.697860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.699220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.710933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 295/576] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:50.711678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.791052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b331b71-c0d5-4b28-a91f-5682e5c0a789 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.818109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7213f2-1db9-4c08-8b69-7dc874328f2c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.826567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcbdd18-0d9b-4b71-b7aa-5bc1577750de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.827246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcbdd18-0d9b-4b71-b7aa-5bc1577750de None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.827531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcbdd18-0d9b-4b71-b7aa-5bc1577750de None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.827866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcbdd18-0d9b-4b71-b7aa-5bc1577750de None admin] PATH_INFO: `/v3/roles/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.838941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcbdd18-0d9b-4b71-b7aa-5bc1577750de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-edcbdd18-0d9b-4b71-b7aa-5bc1577750de None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:50.844134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:50.845061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:50.845898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 282/577] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:50.847185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcbdd18-0d9b-4b71-b7aa-5bc1577750de None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.853635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:50.859829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.860471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.860755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.861078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.863912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:50.872519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:50.874079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:50.889722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:50.890707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:50.891589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:50.891589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:50.891589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:50.891589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 296/578] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:50.892012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:50.960477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcbdd18-0d9b-4b71-b7aa-5bc1577750de None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.994605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365bbf59-f69c-4086-9edd-86da4d0ba444 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:50.997556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:50.998166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:50.998448 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:50.998723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:50.999368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:50.999672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.001060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.018702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 283/579] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:51.018702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.034553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.035185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.035468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.035769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] PATH_INFO: `/v3/users/nova` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.043473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.044961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:51.056834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:51.057775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 297/580] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:56:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:51.058717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.115732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:51.132403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:51.169163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20bad62-945b-4524-a649-5edecff9dc0f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.184541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aaa622-22d6-4b04-b320-b6bb271c60db None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.239790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.239978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.240070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.240070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.240070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.240274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.240441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.247721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.247908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:51.247908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.248085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:51.248085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.248231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.248312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.256079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 284/581] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:51.260503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 298/582] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:51.260645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.260732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.382451 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767666f1-9045-4760-bb8d-d5746564d60d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.386014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:51.404283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74c3d91-924b-40fe-95f1-988191646852 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.428639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.429487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.432443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.432910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.433814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:51.434273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.435553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.442631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.443257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.443534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.443834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.446668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:51.453733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.455223 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.471001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 299/583] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:51.471001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:51.472101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:51.472969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 285/584] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/users/cinder => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:51.473887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.582513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721a25e5-e0b7-43cb-833c-35f50d8d5d9f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.583212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0208cc7e-e9f9-4570-aced-73d17e5f4135 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.617335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.618027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.618312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.618587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.618844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.619459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.619766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.620041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.626599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.628983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.630322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.632269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:51.642152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:51.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:51.644683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:51.644683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:51.644683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:51.644683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:51.644683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:51.644683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:51.644683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:51.644683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:51.645791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 286/585] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:51.646845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:51.648126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:51.649566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:51.650759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:51.650759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:51.650759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:51.650759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 300/586] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:51.650759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.669390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:51.711252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:51.759397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfe60ab-d61a-47b0-a338-9ac86f04f19b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.766324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaf9d3a-cd6d-43b5-9f40-4ce350b7476c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.796678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.797318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.797596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.797875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.804244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.804851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.805131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.805407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.806547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:51.806863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.808825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.809717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.811144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:51.825652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:51.832341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:51.833277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:51.833277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:51.833277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:51.833456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 301/587] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:51.833529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 287/588] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:51.833693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.833765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.905428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:51.935848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc06056b-99a0-4063-be17-4065b697b103 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.942621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057e9562-2bc6-46c6-bef6-f1658b3e1cc0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:51.952272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:51.971643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.972280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.972583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.972827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.973515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:51.973819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.975257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.979614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:51.980240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:51.980520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:51.980792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:51.981428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:51.981731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:51.983070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:51.992284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 302/589] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:51.993302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:51.997280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 288/590] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:51.998278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.098754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673e6f4d-344f-4ed8-b43a-b6ec5fa5b7ef None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.104656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab621040-c6ca-461f-baa5-d9cd6afbebe6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.130206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:52.136352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.136981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.137257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.137525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.143556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.144196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.144475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.144749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.145395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:52.145699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.147062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.149644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.151082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.163554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 289/591] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:52.163554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:52.170577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:52.171493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:52.172509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:52.172509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:52.172509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:52.175925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 303/592] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:52.177361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.196394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:52.280121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1963f164-1577-4c09-88df-b7bfb618d28a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.287330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0850ce-5846-47c4-b6cc-8b3506318307 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.324517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269c1c0e-2fa5-44e8-94af-2b81c5b546dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.325175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269c1c0e-2fa5-44e8-94af-2b81c5b546dc None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.325454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269c1c0e-2fa5-44e8-94af-2b81c5b546dc None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.325726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269c1c0e-2fa5-44e8-94af-2b81c5b546dc None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/54db2e6805fd40f79c03eb3b2319f2b5/roles/91265edc38fd488492ee9ae7ab7d2440` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.327398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.328060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.328346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.328625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.336355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.337797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.342412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269c1c0e-2fa5-44e8-94af-2b81c5b546dc None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=54db2e6805fd40f79c03eb3b2319f2b5, role_id=91265edc38fd488492ee9ae7ab7d2440)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.343848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269c1c0e-2fa5-44e8-94af-2b81c5b546dc None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:52.350086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:52.351204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:52.352574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 304/593] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:52 2022] GET /identity/v3/users/neutron => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:52.353201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.364799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 290/594] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:52 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/54db2e6805fd40f79c03eb3b2319f2b5/roles/91265edc38fd488492ee9ae7ab7d2440 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:52.366013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269c1c0e-2fa5-44e8-94af-2b81c5b546dc None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.404718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:52.470198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998ac0a-8962-4f54-93cb-bec3740b0e07 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.492233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:52.517913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.519314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.519397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.519614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.526393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269c1c0e-2fa5-44e8-94af-2b81c5b546dc None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.529547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.532242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:52.550123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:52.551380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:52.552313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:52.552313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:52.552313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:52.552313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:52.552313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:52.552313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:52.552313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:52.552313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:52.555888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 305/595] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:52.560460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.578062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.578716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.579005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.579281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.579999 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:52.580278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.582452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.597908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 291/596] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:52.598766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.698288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05779cb6-4f8a-4abe-bbb8-f498a8a14f2c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.700576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:52.720563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3ed3de-279e-4df7-b42d-f8d0be858916 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.735522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.736532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.737002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.737267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.738004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:52.738379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.739854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.756517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08277bb-998b-4775-b85d-f799df67ffec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.757122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08277bb-998b-4775-b85d-f799df67ffec None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.762160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08277bb-998b-4775-b85d-f799df67ffec None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.762292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 306/597] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Sep 15 13:56:52 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:52.762402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08277bb-998b-4775-b85d-f799df67ffec None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.762499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.762587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08277bb-998b-4775-b85d-f799df67ffec None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:52.762587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08277bb-998b-4775-b85d-f799df67ffec None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.762587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08277bb-998b-4775-b85d-f799df67ffec None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.771174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:52.781719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 292/598] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:52.782786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf494f0-3833-4a3d-a39f-5eb811c468eb None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.783074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf494f0-3833-4a3d-a39f-5eb811c468eb None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.783409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf494f0-3833-4a3d-a39f-5eb811c468eb None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.786156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 293/599] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:52.786722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf494f0-3833-4a3d-a39f-5eb811c468eb None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.870168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a31e0c-fd26-4185-ae43-b3c69738cf15 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.895482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf494f0-3833-4a3d-a39f-5eb811c468eb None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:52.905695 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.907672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.907672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.907672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.914996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.916781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:52.930691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:52.931665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:52.932683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 307/600] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:52.933207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:52.939936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0495868a-9bf4-4d83-8fb0-48b39cb03cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:52.940710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495868a-9bf4-4d83-8fb0-48b39cb03cbf None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.941095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495868a-9bf4-4d83-8fb0-48b39cb03cbf None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.941470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495868a-9bf4-4d83-8fb0-48b39cb03cbf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.952569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0495868a-9bf4-4d83-8fb0-48b39cb03cbf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:52.954078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0495868a-9bf4-4d83-8fb0-48b39cb03cbf None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:52.964330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-0495868a-9bf4-4d83-8fb0-48b39cb03cbf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:52.973010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:52.974172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:52.975259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:52.975259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:52.975259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:52.980872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 294/601] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:52.980872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a45fa-f3da-46b5-bdf5-15b2d11d3e42 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:52.980872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a45fa-f3da-46b5-bdf5-15b2d11d3e42 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:52.980872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a45fa-f3da-46b5-bdf5-15b2d11d3e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:52.997232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-583a45fa-f3da-46b5-bdf5-15b2d11d3e42 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:52.998548 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-583a45fa-f3da-46b5-bdf5-15b2d11d3e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:53.046552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:53.050580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ac32ca-d276-4ca6-ad17-20e00ff5dd7e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:53.075056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 295/602] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:53.076201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583a45fa-f3da-46b5-bdf5-15b2d11d3e42 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:53.092461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539fb473-e4db-47d9-bbc9-170d715be5ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:53.093364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fb473-e4db-47d9-bbc9-170d715be5ba None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:53.093778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fb473-e4db-47d9-bbc9-170d715be5ba None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:53.094214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fb473-e4db-47d9-bbc9-170d715be5ba None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/ce6744790db74f0aaffc6eb269bf6737/roles/91265edc38fd488492ee9ae7ab7d2440` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:53.109941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539fb473-e4db-47d9-bbc9-170d715be5ba None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=ce6744790db74f0aaffc6eb269bf6737, role_id=91265edc38fd488492ee9ae7ab7d2440)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:53.111611 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539fb473-e4db-47d9-bbc9-170d715be5ba None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:53.134176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 308/603] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:52 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/ce6744790db74f0aaffc6eb269bf6737/roles/91265edc38fd488492ee9ae7ab7d2440 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:53.135268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539fb473-e4db-47d9-bbc9-170d715be5ba None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:53.244302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583a45fa-f3da-46b5-bdf5-15b2d11d3e42 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:53.287898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:53.288520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:53.288914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:53.289190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:53.289942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:53.290457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:53.292527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:53.307366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 296/604] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:53.308578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:53.321386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:53.369745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539fb473-e4db-47d9-bbc9-170d715be5ba None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:53.389478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:53.412143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:53.412765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:53.413044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:53.413317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:53.413966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:53.414267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:53.415716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:53.428503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 309/605] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:53.429330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:53.467510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297441ec-3b64-4e65-a509-a1830763ae73 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:53.504283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d7554-d043-4b96-9b99-289f18f010a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:53.504916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d7554-d043-4b96-9b99-289f18f010a5 None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:53.505127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d7554-d043-4b96-9b99-289f18f010a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:53.505402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d7554-d043-4b96-9b99-289f18f010a5 None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/f54979e322484a42bb57d1b3a619652c/roles/91265edc38fd488492ee9ae7ab7d2440` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:53.521000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d7554-d043-4b96-9b99-289f18f010a5 None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=f54979e322484a42bb57d1b3a619652c, role_id=91265edc38fd488492ee9ae7ab7d2440)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:53.522440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d7554-d043-4b96-9b99-289f18f010a5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:53.538945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 297/606] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:53 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/f54979e322484a42bb57d1b3a619652c/roles/91265edc38fd488492ee9ae7ab7d2440 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:53.544156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d7554-d043-4b96-9b99-289f18f010a5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:53.568697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d724c8f-ba6a-4a2e-bf51-9c4821cd3f69 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:53.607522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:53.628367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85f7470-9b2b-4093-9d02-7d80f200f16a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:53.629705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f7470-9b2b-4093-9d02-7d80f200f16a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:53.630149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f7470-9b2b-4093-9d02-7d80f200f16a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:53.630658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85f7470-9b2b-4093-9d02-7d80f200f16a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:53.650817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85f7470-9b2b-4093-9d02-7d80f200f16a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-c85f7470-9b2b-4093-9d02-7d80f200f16a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:53.663303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:53.664552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:53.665590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:53.668871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 310/607] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:53.678577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85f7470-9b2b-4093-9d02-7d80f200f16a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:53.757721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:53.789281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d7554-d043-4b96-9b99-289f18f010a5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:53.813217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85f7470-9b2b-4093-9d02-7d80f200f16a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:53.831917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:53.832758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:53.833145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:53.833578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:53.841296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:53.843055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:53.852925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:53.853538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:53.853817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:53.854090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:53.854756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:53.855056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:53.856061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:53.856997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:53.857864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:53.857864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:53.857864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:53.857864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:53.857864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:53.857864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:53.857864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:53.857864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:53.858286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:53.858355 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 298/608] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:53.864506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:53.869281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 311/609] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:53.874779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:53.940150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:53.982672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3655bc02-9daa-4904-8e6f-44bf27573743 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:53.994694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73766535-b844-4ce9-ad78-86d9287f2938 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:54.019428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:54.019992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.020273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.020549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.021422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:54.021733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:54.023807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:54.033989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:54.035056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.035527 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.036071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.038168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 299/610] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:54.048754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:54.056773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:54.058622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:54.074296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:54.091647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:54.092661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:54.093542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:54.093542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:54.093542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:54.105118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 312/611] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:54.107346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:54.225602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb8249-946e-4172-b55a-df597ad2519b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:54.295480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853bd2ee-1980-478d-8726-3a751301995d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:54.303374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3136871e-e249-4709-862f-33bffe751eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:54.304457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3136871e-e249-4709-862f-33bffe751eca None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.304956 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3136871e-e249-4709-862f-33bffe751eca None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.305432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3136871e-e249-4709-862f-33bffe751eca None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/922012eacf244f9480c180c2eb9eaf28/roles/91265edc38fd488492ee9ae7ab7d2440` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.318158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:54.326736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3136871e-e249-4709-862f-33bffe751eca None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=922012eacf244f9480c180c2eb9eaf28, role_id=91265edc38fd488492ee9ae7ab7d2440)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:54.328754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3136871e-e249-4709-862f-33bffe751eca None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:54.356008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 300/612] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:54 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/922012eacf244f9480c180c2eb9eaf28/roles/91265edc38fd488492ee9ae7ab7d2440 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:56:54.357631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:54.358478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.358989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.359465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.360384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:54.360950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:54.363835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:54.378258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 313/613] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:54.384865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3136871e-e249-4709-862f-33bffe751eca None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:54.466260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:54.519797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3136871e-e249-4709-862f-33bffe751eca None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:54.562838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45266516-5d01-450e-80c1-1a714b6614df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:54.563393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45266516-5d01-450e-80c1-1a714b6614df None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.563672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45266516-5d01-450e-80c1-1a714b6614df None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.563971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45266516-5d01-450e-80c1-1a714b6614df None admin] PATH_INFO: `/v3/users/swift` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.573324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45266516-5d01-450e-80c1-1a714b6614df None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:54.574783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45266516-5d01-450e-80c1-1a714b6614df None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-45266516-5d01-450e-80c1-1a714b6614df None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:54.588168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:54.589205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:54.590171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 301/614] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:56:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:54.598171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:54.738208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c979d44c-aebb-4970-bb25-923796a2dbd3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:54.780432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:54.781063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.781341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.781614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.782365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:54.782673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:54.784493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:54.797507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230fb0-0d33-41b6-825d-de8f1ac7ea58 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.797800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230fb0-0d33-41b6-825d-de8f1ac7ea58 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.798061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0230fb0-0d33-41b6-825d-de8f1ac7ea58 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.800446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 302/615] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:54.806027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ab2d98-eabe-4baf-a603-19b77b516b98 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.806309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ab2d98-eabe-4baf-a603-19b77b516b98 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.806577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ab2d98-eabe-4baf-a603-19b77b516b98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.807442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 314/616] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 15 13:56:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:54.813099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:54.826212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-78ab2d98-eabe-4baf-a603-19b77b516b98 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:54.827377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-78ab2d98-eabe-4baf-a603-19b77b516b98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:54.852100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:54.924456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec91f3-336c-48c8-b5f4-15fa8e9f1438 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:54.961803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:54.962442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:54.962722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:54.962996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:54.974869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:54.976350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:54.991976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:54.992963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:54.993841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:54.993841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:54.993841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:54.993841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 315/617] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:54.999119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:55.053387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 303/618] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:55.081159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:55.104030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cd0c0-20b0-4edd-a166-b4c9cd5cf456 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:55.139664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:55.140303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:55.140582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:55.140855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:55.141500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:55.141806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:55.143316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:55.159925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 316/619] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:55.161959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ab2d98-eabe-4baf-a603-19b77b516b98 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:55.221122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:55.266001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ab2d98-eabe-4baf-a603-19b77b516b98 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:55.301509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:55.302147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:55.302427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:55.302701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:55.309723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:55.311182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:55.323255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:55.324169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:55.325053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 304/620] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:55.337242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:55.426374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:55.447802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19741d8-26fe-42df-b059-7bb7272e221e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:55.495715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:55.496872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:55.498542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:55.499799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:55.501316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:55.501981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:55.505734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:55.512678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:55.532621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 317/621] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:55.538615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:55.582540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11375bcf-4fe9-4319-852c-76aadee2ce82 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:55.602680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:55.665423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364f4d4-2b1b-469b-853b-397c502eee52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:55.666038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364f4d4-2b1b-469b-853b-397c502eee52 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:55.666318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364f4d4-2b1b-469b-853b-397c502eee52 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:55.666594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364f4d4-2b1b-469b-853b-397c502eee52 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:55.683653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742f20c5-396b-4426-8132-ceb40a6699c9 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:55.700552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6364f4d4-2b1b-469b-853b-397c502eee52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-6364f4d4-2b1b-469b-853b-397c502eee52 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:55.706405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:55.707340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:55.708541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:55.713713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 305/622] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:55.715845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364f4d4-2b1b-469b-853b-397c502eee52 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:55.728567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475bb2fe-a02a-4210-9e98-1bcdefb43de5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:55.729669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475bb2fe-a02a-4210-9e98-1bcdefb43de5 None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:55.730145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475bb2fe-a02a-4210-9e98-1bcdefb43de5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:55.730589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475bb2fe-a02a-4210-9e98-1bcdefb43de5 None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/30fc2a26b4fc4d24b984fa8097752055/roles/91265edc38fd488492ee9ae7ab7d2440` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:55.748528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475bb2fe-a02a-4210-9e98-1bcdefb43de5 None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=30fc2a26b4fc4d24b984fa8097752055, role_id=91265edc38fd488492ee9ae7ab7d2440)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:55.750700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475bb2fe-a02a-4210-9e98-1bcdefb43de5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:55.775548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 318/623] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:56:55 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/30fc2a26b4fc4d24b984fa8097752055/roles/91265edc38fd488492ee9ae7ab7d2440 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 13:56:55.789239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa33426-e064-4bbf-a846-30560f0202b4 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:55.789525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa33426-e064-4bbf-a846-30560f0202b4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:55.789802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa33426-e064-4bbf-a846-30560f0202b4 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:55.791290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 319/624] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:56:55.797010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6806d74-c851-48ba-82b3-1ce84b9bf8da None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:55.797333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6806d74-c851-48ba-82b3-1ce84b9bf8da None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:55.797555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6806d74-c851-48ba-82b3-1ce84b9bf8da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:55.817973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a6806d74-c851-48ba-82b3-1ce84b9bf8da None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:55.819187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a6806d74-c851-48ba-82b3-1ce84b9bf8da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:55.864022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364f4d4-2b1b-469b-853b-397c502eee52 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:55.900460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:55.944007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:55.944631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:55.944913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:55.945187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:55.945863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:55.946166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:55.947545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:55.952602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 320/625] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:56:55.961893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 306/626] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:56:55.967278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6806d74-c851-48ba-82b3-1ce84b9bf8da None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:56.099768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6806d74-c851-48ba-82b3-1ce84b9bf8da None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:56.138332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:56.138893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:56.139176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:56.139448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:56.150458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:56.152098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:56.167907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:56.168884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:56.169775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:56.169775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:56.169775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:56.169775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 321/627] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:56.177282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:56.221657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:56.323740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d7f54a-cd13-41df-b240-ab3980023842 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:56.358807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:56.390358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:56.390898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:56.391174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:56.391444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:56.392148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:56.392516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:56.394634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:56.409436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 307/628] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:56.415331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:56.420833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:56.548083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1376d8-ba47-435d-b030-084ba85041b4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:56.572338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:56.584009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d477fda2-53db-4c55-81d3-0b7518f6552c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:56.593578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54a5801-7cb8-48ba-8afa-e426f0bfa1e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:56.594652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54a5801-7cb8-48ba-8afa-e426f0bfa1e9 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:56.594937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54a5801-7cb8-48ba-8afa-e426f0bfa1e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:56.595208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54a5801-7cb8-48ba-8afa-e426f0bfa1e9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:56.608573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54a5801-7cb8-48ba-8afa-e426f0bfa1e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-e54a5801-7cb8-48ba-8afa-e426f0bfa1e9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:56.617218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:56.618229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:56.619416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 322/629] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:56:56.620316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e76847-337e-4121-b9c5-c239fd7a9391 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:56.620316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e76847-337e-4121-b9c5-c239fd7a9391 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:56.620316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e76847-337e-4121-b9c5-c239fd7a9391 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:56.622743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 323/630] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:56.627134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed680f-320d-4ed1-b231-419ebaa7362c None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:56.627491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed680f-320d-4ed1-b231-419ebaa7362c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:56.627806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed680f-320d-4ed1-b231-419ebaa7362c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:56.631979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:56.632837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:56.633314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:56.633776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] PATH_INFO: `/v3/users/glance` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:56.641730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:56.643734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:56.646269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-13ed680f-320d-4ed1-b231-419ebaa7362c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:56.647460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-13ed680f-320d-4ed1-b231-419ebaa7362c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:56.656723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:56.657658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 15 13:56:56.659391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:56.666636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 308/631] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:56 2022] GET /identity/v3/users/glance => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:56.667824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:56.726549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 324/632] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:56.726549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ed680f-320d-4ed1-b231-419ebaa7362c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:56.750412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:56.829703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5940aeae-6345-4c34-b282-67dda5f52b9c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:56.865657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:56.866252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:56.866533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:56.866807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:56.867511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:56.867851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:56.869440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:56.878667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 309/633] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:56.888773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:56.890789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ed680f-320d-4ed1-b231-419ebaa7362c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:56.906198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:56.932892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:56.933964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:56.934472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:56.934963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:56.935927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:56.936427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:56.938081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:56.973700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 325/634] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 15 13:56:56 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:56.979456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:56.999164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d71369b-346f-4982-a4a1-673a293aa96d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.035953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.036574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.036857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.037131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.048525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:57.049718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:57.064388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:57.065772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:57.066655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:57.066655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:57.066655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:57.066655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 310/635] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:57.071098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:57.101875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:57.128994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261dac5d-ddc6-4c74-ae5a-f7d42c6fbcd1 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.165979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51e6d4d-b29d-4714-b7d8-ab9e82d6e3a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.166924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51e6d4d-b29d-4714-b7d8-ab9e82d6e3a3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.167397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51e6d4d-b29d-4714-b7d8-ab9e82d6e3a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.168065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51e6d4d-b29d-4714-b7d8-ab9e82d6e3a3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.200626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:57.247949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51e6d4d-b29d-4714-b7d8-ab9e82d6e3a3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:57.249688 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51e6d4d-b29d-4714-b7d8-ab9e82d6e3a3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:57.254080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789e032f-fe2e-4a19-88e3-a234193d8134 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.301917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.303297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.304059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.304544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.305679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:57.306677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:57.308860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:57.330474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 311/636] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:57.332485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:57.344261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:57.375289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 326/637] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:56:57.375289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51e6d4d-b29d-4714-b7d8-ab9e82d6e3a3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:57.452869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666796fe-4ec2-4bba-a720-1fd0d727d0ee None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.473916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51e6d4d-b29d-4714-b7d8-ab9e82d6e3a3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.481192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:57.492700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed12a52f-f590-45d8-8c3f-39d15de9a503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.493325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12a52f-f590-45d8-8c3f-39d15de9a503 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.493599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12a52f-f590-45d8-8c3f-39d15de9a503 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.493867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12a52f-f590-45d8-8c3f-39d15de9a503 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.504705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed12a52f-f590-45d8-8c3f-39d15de9a503 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-ed12a52f-f590-45d8-8c3f-39d15de9a503 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:57.509598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:57.510549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:57.511391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 312/638] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:57.512209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69914d0-81bd-4206-9758-984126e53094 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.512209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69914d0-81bd-4206-9758-984126e53094 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.512908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69914d0-81bd-4206-9758-984126e53094 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.518324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 313/639] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:57.518324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69914d0-81bd-4206-9758-984126e53094 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:57.520444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.521087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.521366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.521639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.529735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:57.531179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:57.545942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:57.546898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 327/640] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:57.548488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:57.587360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:57.628781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69914d0-81bd-4206-9758-984126e53094 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.659334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4d0c5f-65a2-4a6e-92d2-72b9f2632ad2 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.667124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fcf65b-9ec5-422c-b151-5ad8b99aa34e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.668043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fcf65b-9ec5-422c-b151-5ad8b99aa34e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.668828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fcf65b-9ec5-422c-b151-5ad8b99aa34e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.669325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fcf65b-9ec5-422c-b151-5ad8b99aa34e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.681706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fcf65b-9ec5-422c-b151-5ad8b99aa34e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:57.683784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fcf65b-9ec5-422c-b151-5ad8b99aa34e None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:57.694502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.695109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.695384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.695649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.696342 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:57.696639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:57.697986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-21fcf65b-9ec5-422c-b151-5ad8b99aa34e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:57.699981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:57.700967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:57.701870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:57.701870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:57.701870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:57.701870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 314/641] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:57.702861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce14f2-5897-4df2-a25f-0e912662ca52 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.702861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce14f2-5897-4df2-a25f-0e912662ca52 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.702861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fce14f2-5897-4df2-a25f-0e912662ca52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.712931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 328/642] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:57.717145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:57.722539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-7fce14f2-5897-4df2-a25f-0e912662ca52 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:57.723765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7fce14f2-5897-4df2-a25f-0e912662ca52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:57.755266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:57.799619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 315/643] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:57.800044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fce14f2-5897-4df2-a25f-0e912662ca52 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:57.831523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:57.837595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11766d22-21f3-41e4-a7ac-fe094fd484a0 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.883650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.885618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.886083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.886530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.887436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:57.887967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:57.889570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:57.918540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 329/644] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:57.919534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:57.921054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fce14f2-5897-4df2-a25f-0e912662ca52 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:57.963656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:57.964634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:57.965112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:57.965574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:57.966424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:57.966906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:57.968754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:57.987970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 316/645] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:57.988973 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.024176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:58.029517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd42aed-7100-4567-8243-308bc3f46ab9 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.069133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.069806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.070086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.070360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.081137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.082585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:58.097298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:58.098382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:58.099443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:58.099443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:58.099443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:58.099443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 330/646] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:58.099707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.111432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:58.117250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06965f5-5e8a-4a9b-9f0d-316492c78829 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.156160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c0a0f4-661b-443d-bcf3-4c0eb1f461e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.156787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c0a0f4-661b-443d-bcf3-4c0eb1f461e6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.157074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c0a0f4-661b-443d-bcf3-4c0eb1f461e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.157345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c0a0f4-661b-443d-bcf3-4c0eb1f461e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.211314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63122a41-eb2b-4aaf-b766-be510b2e398c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.232838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c0a0f4-661b-443d-bcf3-4c0eb1f461e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.234327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c0a0f4-661b-443d-bcf3-4c0eb1f461e6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.249967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.250596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.250882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.251145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.258203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.259638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.263080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:58.271292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:58.272369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 331/647] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:58.273313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.341126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 317/648] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:58.341921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:58.342038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c0a0f4-661b-443d-bcf3-4c0eb1f461e6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.379942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebae2172-8ab7-46d8-86dc-2fabc5f67da5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.413083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6b4a33-fcd9-4a36-b541-df9c48abda68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.413711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b4a33-fcd9-4a36-b541-df9c48abda68 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.413990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b4a33-fcd9-4a36-b541-df9c48abda68 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.414261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6b4a33-fcd9-4a36-b541-df9c48abda68 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.424381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6b4a33-fcd9-4a36-b541-df9c48abda68 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-bd6b4a33-fcd9-4a36-b541-df9c48abda68 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:58.429868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:58.430836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:58.431678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 332/649] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:58.432410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb133baf-5d57-4710-8ccb-b9ed285cfe43 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.432410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb133baf-5d57-4710-8ccb-b9ed285cfe43 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.436395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb133baf-5d57-4710-8ccb-b9ed285cfe43 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.436395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 333/650] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:56:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:56:58.436395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb133baf-5d57-4710-8ccb-b9ed285cfe43 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.449070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c0a0f4-661b-443d-bcf3-4c0eb1f461e6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.477140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:58.485325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.485872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.486147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.486416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.487865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:58.488172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.489542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.500685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 318/651] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:58.501467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.540570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb133baf-5d57-4710-8ccb-b9ed285cfe43 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.566951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:58.574742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.575345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.575621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.575919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.576604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:58.576903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.578315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.590628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 334/652] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:58.591411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.602396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe4f06c-6f09-4be6-a7e9-44e10687599a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.636195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d84693-37b3-4002-b091-4efab177195e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.636796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d84693-37b3-4002-b091-4efab177195e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.637081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d84693-37b3-4002-b091-4efab177195e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.637353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d84693-37b3-4002-b091-4efab177195e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.647762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d84693-37b3-4002-b091-4efab177195e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.649203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d84693-37b3-4002-b091-4efab177195e None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-62d84693-37b3-4002-b091-4efab177195e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:58.663184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:58.664178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:58.668609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:58.668609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:58.668609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:58.668609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 319/653] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:58.668609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5395d-c52f-4ed1-a060-76932e7faaab None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.668609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5395d-c52f-4ed1-a060-76932e7faaab None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.668609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5395d-c52f-4ed1-a060-76932e7faaab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.685077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-86d5395d-c52f-4ed1-a060-76932e7faaab None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:56:58.686253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-86d5395d-c52f-4ed1-a060-76932e7faaab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:56:58.696086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cbbea8-0f32-4cbd-824d-067eba3301e0 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.720278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:58.729668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.730272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.730547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.730817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.731466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:58.731791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.733152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.741507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 335/654] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:58.742228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.753480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 320/655] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:56:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:56:58.754312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d5395d-c52f-4ed1-a060-76932e7faaab None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.796787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:58.853719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522146c-3f3d-4be7-ac66-f48c74e442d1 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.867678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d5395d-c52f-4ed1-a060-76932e7faaab None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:58.889179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.889807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.890206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.890473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.899096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.900624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.902346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:58.902941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:58.903229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:58.903504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:58.904265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=54db2e6805fd40f79c03eb3b2319f2b5) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:58.904577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:58.905982 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:58.912157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 321/656] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/role_assignments?user.id=54db2e6805fd40f79c03eb3b2319f2b5&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:58.912885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:58.913799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 15 13:56:58.914738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:58.915289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.922124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 336/657] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/users/cinder => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:58.922124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:58.970652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.057261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4826d05-093e-4435-9b4e-d606ca3ff007 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.063167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99cbc71-6b19-4a1c-b84c-d50f6df63dbd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.077459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.095770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.097036 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.097314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.097569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.098385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:59.099356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.100772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.104360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.104640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.105996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.105996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.113071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 322/658] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:59.113850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.119398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.120213 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:59.137700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:59.138877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:56:59.140019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:56:59.140019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:59.140019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:59.140019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 337/659] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:59.140019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.230633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.239518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7ffd04-447f-44c0-b6a8-abec964d9d9c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.254289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30367d-c3db-4953-931f-f2feca2198ae None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.274986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.275809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.276168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.276517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.277326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:59.277731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.279221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.291898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f41dfb2-c555-4b4a-9478-ccba91b0620c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.292528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f41dfb2-c555-4b4a-9478-ccba91b0620c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.292803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f41dfb2-c555-4b4a-9478-ccba91b0620c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.293068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f41dfb2-c555-4b4a-9478-ccba91b0620c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.303278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 323/660] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:59.304354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.307788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f41dfb2-c555-4b4a-9478-ccba91b0620c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-4f41dfb2-c555-4b4a-9478-ccba91b0620c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:56:59.315025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:59.316021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 338/661] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:59.317007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f41dfb2-c555-4b4a-9478-ccba91b0620c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.343068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.416105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aede621-2947-4b9a-ac1f-d7ff6f386cc5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.425151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f41dfb2-c555-4b4a-9478-ccba91b0620c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.451901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.452523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.452798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.453066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.460580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.461138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.461929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.462058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.462463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.462875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.463669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:59.464002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.465350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:56:59.473591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:59.474479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 324/662] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:59.475465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.487638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 339/663] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:59.487638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.492156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.572147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.583763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c80cc4-95b7-445b-a840-fd636d4c1c61 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.592594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6314d22-b1b7-4b30-bddb-35d0abc4ff47 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.619198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce10e425-2fc0-4631-b81f-4fbcbbb40f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.619930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce10e425-2fc0-4631-b81f-4fbcbbb40f4f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.620215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce10e425-2fc0-4631-b81f-4fbcbbb40f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.620495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce10e425-2fc0-4631-b81f-4fbcbbb40f4f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.629878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.630479 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.630753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.631026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.631678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:59.632013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.633375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.644357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 340/664] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:59.645184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.695321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce10e425-2fc0-4631-b81f-4fbcbbb40f4f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.696910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce10e425-2fc0-4631-b81f-4fbcbbb40f4f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.711032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.753498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac225ec-c31e-42b0-b520-e741f3b9719f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.788721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.789329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.789613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.789890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.790580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:56:59.790897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.792389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.810150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 341/665] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:56:59.810328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 325/666] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:56:59.810328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce10e425-2fc0-4631-b81f-4fbcbbb40f4f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.810967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.816091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.916855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4960ab83-f74b-406a-a9e9-74d94044cc1c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.927171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce10e425-2fc0-4631-b81f-4fbcbbb40f4f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:56:59.942748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:56:59.955190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.956188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.956475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.956815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.963913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.965349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.965520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:56:59.966136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:56:59.966410 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:56:59.966677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:56:59.977072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:56:59.978467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:56:59.979012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:59.980145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 342/667] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/users/neutron => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:56:59.981147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:56:59.997976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:56:59.999115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:00.000228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:00.000228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:00.000228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:00.002173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 326/668] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:00.003423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.057390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:00.097045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d9d919-6212-4bbf-ba97-0b9c71bde638 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.113361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20911332-a007-4717-a56c-a96eeef8484b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.132817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.134060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.134493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.135163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.136226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=ce6744790db74f0aaffc6eb269bf6737) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:00.137028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.139070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.147790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.148513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.148856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.149195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.155730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 343/669] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:56:59 2022] GET /identity/v3/role_assignments?user.id=ce6744790db74f0aaffc6eb269bf6737&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:00.156580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.161352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.163346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:00.185664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:00.186675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:00.187581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:00.187581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:00.187581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:00.191855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 327/670] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:00.193213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.223049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:00.319299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30b6a70-eee5-44ac-a898-2a45247a7a0c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.330800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3238ee-60e5-4d0e-b2fb-1d7f409354ad None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.358498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:00.371409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.372066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.372349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.372625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.373479 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:00.373896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.374728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.375353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.375454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.375641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.375992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.377499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:00.377499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.380007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.395716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 328/671] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:00.395716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.400394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 344/672] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:00.401178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.494964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:00.521495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea71006-d200-4641-8c84-ec039ce52e65 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.528080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece14a59-0eea-4db2-bf35-940db90bc0fa None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.558337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.558977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.559258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.559535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.567668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.568721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.569141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.569331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.569599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.569870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.570509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:00.571082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.573059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:00.581833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:00.582814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:00.583791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 345/673] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:00.584395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.585775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 329/674] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:00.591129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.636969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:00.697871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7b3471-1e0d-42d8-919b-dbba7c7a3565 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.716253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e32f57d-147d-4fe6-b8fd-4535762fca7a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.735885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a91592-f453-400e-943c-aeab5fc230b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.737714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a91592-f453-400e-943c-aeab5fc230b7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.738193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a91592-f453-400e-943c-aeab5fc230b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.738682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a91592-f453-400e-943c-aeab5fc230b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.760788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.761339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.761622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.761899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.765449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:00.773471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.775600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:00.790500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:00.791454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:00.792373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:00.792373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:00.792373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:00.792373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:00.792373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:00.792373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:00.792373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:00.792373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:00.795766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 330/675] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:00.796786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.825319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a91592-f453-400e-943c-aeab5fc230b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.827097 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a91592-f453-400e-943c-aeab5fc230b7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.874155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:00.914313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe584a55-3e8c-4658-9d6c-8ae77e39d264 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:00.938191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 346/676] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:57:00.940018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a91592-f453-400e-943c-aeab5fc230b7 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:00.951890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:00.952494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:00.952770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:00.953041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:00.953871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:00.954178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:00.956294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:00.979740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 331/677] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:00.979740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:01.044134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:01.050170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a91592-f453-400e-943c-aeab5fc230b7 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:01.086171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a332c-1837-4f1a-9000-c89d0cc95a7f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:01.087600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:01.088521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.088899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.089420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.090230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:01.090647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:01.092273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:01.106531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 347/678] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:01.107561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:01.117771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:01.129500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:01.130422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.130897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.131308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.142050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:01.144196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:01.160470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:01.161453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:01.162326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:01.162326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:01.162326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:01.164452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 332/679] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:01.165530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:01.224009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7b0c2c-41db-4fd3-9e85-985bfe5f7923 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:01.261531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccd2f03-e95c-466c-82a7-5bdb0e3074f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:01.262140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd2f03-e95c-466c-82a7-5bdb0e3074f9 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.262418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd2f03-e95c-466c-82a7-5bdb0e3074f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.262686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd2f03-e95c-466c-82a7-5bdb0e3074f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.272569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b71150-bc37-40fa-a3c1-4e3a3883bc40 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:01.280010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:01.317950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:01.318561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.318838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.319282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.320335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=f54979e322484a42bb57d1b3a619652c) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:01.320647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:01.322060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:01.336053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 333/680] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:01 2022] GET /identity/v3/role_assignments?user.id=f54979e322484a42bb57d1b3a619652c&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:01.336053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:01.339378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccd2f03-e95c-466c-82a7-5bdb0e3074f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:01.341128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccd2f03-e95c-466c-82a7-5bdb0e3074f9 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:01.350761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:01.460948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 348/681] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:57:01.462965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f074ceb4-ab50-4665-8988-fa6663d131c4 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.463357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f074ceb4-ab50-4665-8988-fa6663d131c4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.463713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f074ceb4-ab50-4665-8988-fa6663d131c4 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.469591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 349/682] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:01 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:01.470465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f074ceb4-ab50-4665-8988-fa6663d131c4 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:01.486014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7efba89-7bc8-423d-bc6d-e3827f163fb3 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:01.529678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d79633-d02d-4beb-b75f-1c5f5b4c147e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:01.531422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d79633-d02d-4beb-b75f-1c5f5b4c147e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.531422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d79633-d02d-4beb-b75f-1c5f5b4c147e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.531422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d79633-d02d-4beb-b75f-1c5f5b4c147e None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.531937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d79633-d02d-4beb-b75f-1c5f5b4c147e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:01.532240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d79633-d02d-4beb-b75f-1c5f5b4c147e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:01.534223 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d79633-d02d-4beb-b75f-1c5f5b4c147e None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:01.549589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 334/683] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:01.549979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229aacba-ecb4-4cd9-824b-b11ee6668619 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.550260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229aacba-ecb4-4cd9-824b-b11ee6668619 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.550533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229aacba-ecb4-4cd9-824b-b11ee6668619 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.572226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-229aacba-ecb4-4cd9-824b-b11ee6668619 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:01.574373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-229aacba-ecb4-4cd9-824b-b11ee6668619 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:01.605828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:01.608645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f074ceb4-ab50-4665-8988-fa6663d131c4 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:01.620140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:01.648744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:01.649798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.650333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.650606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.660097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 335/684] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:57:01.660993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229aacba-ecb4-4cd9-824b-b11ee6668619 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:01.665728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:01.667186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:01.697382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:01.698387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:01.699282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:01.699282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:01.699282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:01.700005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 350/685] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:01.704101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:01.810924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229aacba-ecb4-4cd9-824b-b11ee6668619 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:01.848789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:01.849339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.849614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.849881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.857059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:01.858750 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:01.861142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b86a7e8-16ca-4575-b542-bd134df974c3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:01.875142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:01.876135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:01.878305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 336/686] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:01.883276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:01.934685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:01.935299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:01.936103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:01.936457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:01.937974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:01.938642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:01.939178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:01.940939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:01.970231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 351/687] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:01.977703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:02.014472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:02.047739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0c972e-aeb3-4521-9e0d-bd2fa046ba81 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:02.090823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:02.091559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.091886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.092161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.093074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:02.093276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:02.094839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:02.103168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a1e12-80da-4e3a-9119-60736601199b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:02.110551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 337/688] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:02.112047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:02.141646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:02.142605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.142847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.143034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.150892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:02.152349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:02.165254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:02.166187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:02.167100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 352/689] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:02.168160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:02.213687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:02.240462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373d3e18-a143-43ed-8fbf-fd490702445a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:02.259202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:02.281166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:02.282148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.282626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.283095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.296722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:02.297106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72e6414-a7d3-413d-a2d6-84c0d1e60170 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:02.298803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:02.317360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:02.318404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:02.319822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:02.319822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:02.319822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:02.322005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 338/690] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:02.322990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:02.342575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:02.343254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.343460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.343812 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.344529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=922012eacf244f9480c180c2eb9eaf28) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:02.344894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:02.347367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:02.354262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 353/691] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:02 2022] GET /identity/v3/role_assignments?user.id=922012eacf244f9480c180c2eb9eaf28&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 608 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:02.355179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:02.493185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d593723f-732c-4bb0-a1b1-0438346b3e3b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:02.548201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cf5916-5e5c-4bff-87c9-9850c884b57d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:02.555842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:02.571878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:02.572892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.575157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.575759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.578595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:02.579203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:02.582543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:02.601013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 339/692] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:02.607006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:02.611713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:02.612095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.612325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.612717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.613370 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:02.613677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:02.615055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:02.629124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 354/693] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:02.635264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:02.638920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:02.740859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8092f5f3-d2aa-4b8e-9731-68ac101a106f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:02.764061 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165f752-86ba-4c12-a9ec-d36aa64d8986 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:02.777963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86a8628-a30d-4145-8c7a-41e9d599b747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:02.778698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86a8628-a30d-4145-8c7a-41e9d599b747 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.779130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86a8628-a30d-4145-8c7a-41e9d599b747 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.779494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86a8628-a30d-4145-8c7a-41e9d599b747 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.780444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86a8628-a30d-4145-8c7a-41e9d599b747 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=30fc2a26b4fc4d24b984fa8097752055) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:02.781011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86a8628-a30d-4145-8c7a-41e9d599b747 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:02.782631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86a8628-a30d-4145-8c7a-41e9d599b747 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:02.790009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 340/694] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:02 2022] GET /identity/v3/role_assignments?user.id=30fc2a26b4fc4d24b984fa8097752055&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:02.791462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc7a84-d804-4f26-893b-ef427b6d3a67 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.791846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc7a84-d804-4f26-893b-ef427b6d3a67 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.792210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc7a84-d804-4f26-893b-ef427b6d3a67 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.793896 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 341/695] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:02.800497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77367d46-6953-4d40-822d-2b7dba6b784a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.800904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77367d46-6953-4d40-822d-2b7dba6b784a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.801504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77367d46-6953-4d40-822d-2b7dba6b784a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.803873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509e8acc-0b18-4081-8f78-11b9f877d2a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:02.804581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e8acc-0b18-4081-8f78-11b9f877d2a8 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:02.804938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e8acc-0b18-4081-8f78-11b9f877d2a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:02.805284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e8acc-0b18-4081-8f78-11b9f877d2a8 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:02.806011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e8acc-0b18-4081-8f78-11b9f877d2a8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:02.807537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e8acc-0b18-4081-8f78-11b9f877d2a8 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:02.832095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-77367d46-6953-4d40-822d-2b7dba6b784a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:02.833317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-77367d46-6953-4d40-822d-2b7dba6b784a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:02.836153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 355/696] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:57:02 2022] POST /identity/v3/users => generated 254 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:02.944342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:02.968501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 342/697] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:57:03.261807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509e8acc-0b18-4081-8f78-11b9f877d2a8 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:03.347513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:03.393554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509e8acc-0b18-4081-8f78-11b9f877d2a8 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:03.430127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f68cbd6-d2a6-4f1d-85aa-ca757be23ea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:03.430768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68cbd6-d2a6-4f1d-85aa-ca757be23ea4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:03.431051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68cbd6-d2a6-4f1d-85aa-ca757be23ea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:03.431323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68cbd6-d2a6-4f1d-85aa-ca757be23ea4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:03.442303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f68cbd6-d2a6-4f1d-85aa-ca757be23ea4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-9f68cbd6-d2a6-4f1d-85aa-ca757be23ea4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:03.447726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:03.448636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:03.449665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 356/698] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:03.457092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77367d46-6953-4d40-822d-2b7dba6b784a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:03.593437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77367d46-6953-4d40-822d-2b7dba6b784a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:03.644426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:03.645328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:03.645756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:03.646176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:03.646996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:03.647462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:03.648992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:03.658172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 343/699] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:03.664393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f68cbd6-d2a6-4f1d-85aa-ca757be23ea4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:03.782913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f68cbd6-d2a6-4f1d-85aa-ca757be23ea4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:03.812771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:03.835632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc1bf34-3acb-443f-802a-502d3ded6f58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:03.836285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc1bf34-3acb-443f-802a-502d3ded6f58 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:03.836560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc1bf34-3acb-443f-802a-502d3ded6f58 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:03.836830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc1bf34-3acb-443f-802a-502d3ded6f58 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:03.853462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc1bf34-3acb-443f-802a-502d3ded6f58 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:03.854902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc1bf34-3acb-443f-802a-502d3ded6f58 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-5fc1bf34-3acb-443f-802a-502d3ded6f58 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:03.875494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:03.876481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:03.878304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:03.878304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:03.878304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:03.878304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 357/700] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:03.882272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:03.925753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b86cb-c5c4-4079-90c5-a7524d60c4bc None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:03.926079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b86cb-c5c4-4079-90c5-a7524d60c4bc None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:03.926558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b86cb-c5c4-4079-90c5-a7524d60c4bc None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:03.928200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 358/701] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:03.933378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50294a-eeb0-408a-87b3-a97009e07660 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:03.933659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50294a-eeb0-408a-87b3-a97009e07660 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:03.933931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50294a-eeb0-408a-87b3-a97009e07660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:03.952703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2e50294a-eeb0-408a-87b3-a97009e07660 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:03.953881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2e50294a-eeb0-408a-87b3-a97009e07660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:03.980119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:04.058073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c2752c-4da3-48f5-9263-9c3baf6ef0b1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:04.093068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:04.093635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:04.093873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 359/702] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:57:04.093991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:04.094162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:04.094781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:04.095083 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:04.103955 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:04.120780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 344/703] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:04.125860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e50294a-eeb0-408a-87b3-a97009e07660 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:04.256367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:04.260610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e50294a-eeb0-408a-87b3-a97009e07660 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:04.309257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:04.309893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:04.310148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:04.310424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:04.322063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:04.324090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:04.336615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:04.337841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:04.339033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:04.341142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 360/704] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:57:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:04.346872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:04.462300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:04.464289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211b10e-e42f-459d-a4bc-a033a45e541c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:04.474781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:04.523209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:04.524098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:04.524519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:04.524935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:04.525835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:04.527006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:04.528614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:04.560780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 345/705] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:57:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:04.567117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:04.627207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92066cc5-8257-4920-8816-1efecfead040 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:04.693292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:04.694404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:04.695807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:04.696374 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:04.698276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:04.700124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:04.722042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:04.723267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:04.724313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:04.729064 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 361/706] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Sep 15 13:57:04 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:04.736078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:04.761056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26adabd2-ec72-43e8-a703-1fe95d95f6e3 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:04.784211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:04.850199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9046b17a-06e0-48b3-9797-68aa7c055faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:04.850822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9046b17a-06e0-48b3-9797-68aa7c055faf None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:04.851104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9046b17a-06e0-48b3-9797-68aa7c055faf None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:04.851379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9046b17a-06e0-48b3-9797-68aa7c055faf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:04.868537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef310c9-8cac-4dc6-8423-a6a6bb8fd1ad None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:04.903835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:04.904532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:04.904814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:04.905210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:04.905930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:04.906241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:04.907622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:04.912635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 362/707] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 15 13:57:04 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:04.919128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:04.923440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:04.985071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9046b17a-06e0-48b3-9797-68aa7c055faf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:04.986994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9046b17a-06e0-48b3-9797-68aa7c055faf None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:05.088076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:05.089199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 346/708] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 522 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 15 13:57:05.096547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9046b17a-06e0-48b3-9797-68aa7c055faf None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:05.113265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39fa484-7b6c-403b-893a-28807976fa42 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:05.164063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:05.213216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:05.213855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.214138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.214419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.215077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:05.215383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:05.216846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:05.229139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9046b17a-06e0-48b3-9797-68aa7c055faf None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:05.231116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 363/709] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 15 13:57:04 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:05.296828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb4883f-7d86-4986-84bc-81493ae0cf9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:05.297404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4883f-7d86-4986-84bc-81493ae0cf9d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.297680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4883f-7d86-4986-84bc-81493ae0cf9d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.297951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4883f-7d86-4986-84bc-81493ae0cf9d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.316905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb4883f-7d86-4986-84bc-81493ae0cf9d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:05.318446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb4883f-7d86-4986-84bc-81493ae0cf9d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-adb4883f-7d86-4986-84bc-81493ae0cf9d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:05.342336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:05.343418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:05.344371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:05.344371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:05.344371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:05.344371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 347/710] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:05.349739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:05.401355 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4ed3f9-0300-46f2-b303-2899e4e4dd97 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.401705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4ed3f9-0300-46f2-b303-2899e4e4dd97 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.401901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4ed3f9-0300-46f2-b303-2899e4e4dd97 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.403252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 348/711] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:05.411019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e418ceb8-5aa7-48c7-ae23-82bb83545b1a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.411325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e418ceb8-5aa7-48c7-ae23-82bb83545b1a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.411558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e418ceb8-5aa7-48c7-ae23-82bb83545b1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.437479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e418ceb8-5aa7-48c7-ae23-82bb83545b1a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:05.439060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e418ceb8-5aa7-48c7-ae23-82bb83545b1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:05.496419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321211ea-7c99-4b10-bec0-c2dacdbb9b08 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:05.539777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e596bb-c4ef-4958-874a-e565e0865284 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:05.540007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 349/712] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:57:05.540395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e596bb-c4ef-4958-874a-e565e0865284 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.540679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e596bb-c4ef-4958-874a-e565e0865284 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.541058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e596bb-c4ef-4958-874a-e565e0865284 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.541720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e596bb-c4ef-4958-874a-e565e0865284 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:05.542024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e596bb-c4ef-4958-874a-e565e0865284 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:05.543396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e596bb-c4ef-4958-874a-e565e0865284 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:05.559920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 364/713] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:05.566201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e418ceb8-5aa7-48c7-ae23-82bb83545b1a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:05.626323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348bf45b-77e0-4f96-9ef5-2a6230eec8f0 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.626943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348bf45b-77e0-4f96-9ef5-2a6230eec8f0 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.628045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348bf45b-77e0-4f96-9ef5-2a6230eec8f0 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.629913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 365/714] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:05.636054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2d07c-8df5-4916-bb68-eac8d4ea5ba9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.636615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2d07c-8df5-4916-bb68-eac8d4ea5ba9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.637099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2d07c-8df5-4916-bb68-eac8d4ea5ba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.647896 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:05.686828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-97c2d07c-8df5-4916-bb68-eac8d4ea5ba9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:05.688377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-97c2d07c-8df5-4916-bb68-eac8d4ea5ba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:05.700851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e418ceb8-5aa7-48c7-ae23-82bb83545b1a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:05.705129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:05.741402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0264cd-0167-4517-9eaf-f4e77276965b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:05.742585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0264cd-0167-4517-9eaf-f4e77276965b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.743016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0264cd-0167-4517-9eaf-f4e77276965b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.743435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0264cd-0167-4517-9eaf-f4e77276965b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.753419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0264cd-0167-4517-9eaf-f4e77276965b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:05.755406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0264cd-0167-4517-9eaf-f4e77276965b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-1a0264cd-0167-4517-9eaf-f4e77276965b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:05.770107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:05.771241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:05.772367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:05.772367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:05.772367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:05.772367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:05.772367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:05.772367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:05.772367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:05.772367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:05.775049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 350/715] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:05.776914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed65da5-c170-4d70-938f-6d9145dd6bbd None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.777402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed65da5-c170-4d70-938f-6d9145dd6bbd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.777857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed65da5-c170-4d70-938f-6d9145dd6bbd None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.779519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 351/716] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:05.780417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed65da5-c170-4d70-938f-6d9145dd6bbd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:05.805565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 366/717] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 15 13:57:05.807133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e0e766-af88-4ab6-aacc-0e95d7310313 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.807492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e0e766-af88-4ab6-aacc-0e95d7310313 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.808463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e0e766-af88-4ab6-aacc-0e95d7310313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.833276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-38e0e766-af88-4ab6-aacc-0e95d7310313 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:05.834748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-38e0e766-af88-4ab6-aacc-0e95d7310313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:05.920983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed65da5-c170-4d70-938f-6d9145dd6bbd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:05.938418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:05.950379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 367/718] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:57:05.985495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e0e766-af88-4ab6-aacc-0e95d7310313 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:05.987567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:05.988212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:05.988514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:05.988776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:05.989477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:05.989788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:05.991468 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:06.015326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 352/719] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:06.024436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.114182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:06.166047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e0e766-af88-4ab6-aacc-0e95d7310313 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:06.183010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c921e8-38c2-4e7c-9861-b4789cb82695 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:06.204136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:06.205042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:06.206117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:06.206604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] PATH_INFO: `/v3/services/network` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:06.207389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:06.209100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:06.218620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:06.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:06.220471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:06.221329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:06.221429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:06.221794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:06.222231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:06.223116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=ce6744790db74f0aaffc6eb269bf6737) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:06.223634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 368/720] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:05 2022] GET /identity/v3/services/network => generated 88 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:06.224756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.228060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:06.230058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:06.237078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 353/721] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/role_assignments?user.id=ce6744790db74f0aaffc6eb269bf6737&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 259 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:06.238162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.292881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:06.384759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf3ffdf-7afd-41a4-874e-fa8fae3000d4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:06.406007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f717ba13-e04e-420c-9b29-bc9a78864101 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:06.428175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:06.429167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:06.429541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:06.429822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:06.430614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] RBAC: Adding query filter params (name=network) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:06.430921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:06.432338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:06.437519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 354/722] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/services?name=network => generated 124 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:06.438320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.453262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c15481-269a-4501-94a0-1a3e087a5f47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:06.453903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c15481-269a-4501-94a0-1a3e087a5f47 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:06.454186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c15481-269a-4501-94a0-1a3e087a5f47 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:06.454458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c15481-269a-4501-94a0-1a3e087a5f47 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:06.457846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:06.467092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c15481-269a-4501-94a0-1a3e087a5f47 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-31c15481-269a-4501-94a0-1a3e087a5f47 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:06.473974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:06.474932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:06.475807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 369/723] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:06.476530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c15481-269a-4501-94a0-1a3e087a5f47 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.569067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:06.575916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713a3fbd-ebd2-4361-8abd-6aade3109371 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:06.607079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c15481-269a-4501-94a0-1a3e087a5f47 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:06.613716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e16b7a-fb26-442d-8d03-a0a19f7e6fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:06.614357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e16b7a-fb26-442d-8d03-a0a19f7e6fdd None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:06.614640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e16b7a-fb26-442d-8d03-a0a19f7e6fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:06.614917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e16b7a-fb26-442d-8d03-a0a19f7e6fdd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:06.625607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e16b7a-fb26-442d-8d03-a0a19f7e6fdd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-a9e16b7a-fb26-442d-8d03-a0a19f7e6fdd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:06.631187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:06.632238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:06.633168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 355/724] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:06.633867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e16b7a-fb26-442d-8d03-a0a19f7e6fdd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.648872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:06.649874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:06.650416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:06.650884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:06.651759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] RBAC: Adding query filter params (type=network) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:06.652317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:06.653901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:06.660054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 370/725] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/services?type=network => generated 124 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:06.661090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.729320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:06.804906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590fc7c2-6b48-436f-9d4a-6c4f6793ff89 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:06.859052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e16b7a-fb26-442d-8d03-a0a19f7e6fdd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:06.869754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:06.870705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:06.871187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:06.871655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:06.872644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:06.873273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:06.874949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:06.894426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 371/726] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:06.895727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:06.895727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:06.895958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:06.896137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:06.896758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:06.897066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:06.898526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:06.902333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.921326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 356/727] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:06.928180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:06.952668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:07.074794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b3c1f6-40c1-440e-a004-af18acba7074 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:07.079117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea249782-c8d9-4483-bc59-a7f40ed46ccb None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:07.112560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:07.117219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:07.117926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.118251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.118536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.140606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:07.142055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:07.150748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:07.151386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.151662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.151953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:07.173031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:07.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:07.175031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:07.175031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:07.175031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:07.175201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:07.175738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:07.177432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 372/728] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:07.183048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:07.203666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:07.204683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:07.205708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:07.205708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:07.205708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:07.207785 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 357/729] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:07.210621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:07.327632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:07.383513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe68a2a3-4a7f-47c3-9c2f-51f83d745ff1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:07.420761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd26e40e-4998-4be2-a1a8-f3b906439d3c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:07.424453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:07.425515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.425997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.426733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.427575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:07.428093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:07.429654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:07.444128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 358/730] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:07.449153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:07.496102 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:07.497089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.497570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.498195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.500974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:07.501599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:07.503513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:07.527734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 373/731] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:07.535406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:07.575066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:07.599182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624c651-c5c3-461f-a7ee-870d06a3f4fb None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:07.630526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:07.638016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:07.638655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.638941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.639216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.647234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ca7555-0efb-48db-b81b-e5bc50d825d0 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:07.648203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:07.649753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:07.662090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:07.663073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:07.664045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 359/732] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 13:57:07 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:07.669577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:07.692663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:07.693634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.694136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.694622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.702581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:07.704411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:07.717558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:07.718660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:57:07.719613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:07.722741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 374/733] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:07 2022] GET /identity/v3/users/swift => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:07.736820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:07.787147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4b9106-c754-49b2-a1da-acb6dfc82675 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:07.824249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd011b11-b457-4595-89f7-80dcf61d9c15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:07.824621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd011b11-b457-4595-89f7-80dcf61d9c15 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.824897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd011b11-b457-4595-89f7-80dcf61d9c15 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.825177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd011b11-b457-4595-89f7-80dcf61d9c15 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.825913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd011b11-b457-4595-89f7-80dcf61d9c15 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:07.826278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd011b11-b457-4595-89f7-80dcf61d9c15 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:07.827644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd011b11-b457-4595-89f7-80dcf61d9c15 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:07.839400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:07.845631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb503110-7f8f-41e5-a4dc-e20d3ccffb65 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:07.853517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 360/734] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 15 13:57:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:07.854946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06efcd0d-85e7-4dd5-8a6b-209b52d28587 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.855233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06efcd0d-85e7-4dd5-8a6b-209b52d28587 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.855509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06efcd0d-85e7-4dd5-8a6b-209b52d28587 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.857141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 361/735] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:07.857851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06efcd0d-85e7-4dd5-8a6b-209b52d28587 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:07.884787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e5f35c-1832-4e0d-b1c3-62d8213002df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:07.885749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e5f35c-1832-4e0d-b1c3-62d8213002df None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.886292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e5f35c-1832-4e0d-b1c3-62d8213002df None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.886776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e5f35c-1832-4e0d-b1c3-62d8213002df None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.887815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e5f35c-1832-4e0d-b1c3-62d8213002df None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:07.888322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e5f35c-1832-4e0d-b1c3-62d8213002df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:07.889941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e5f35c-1832-4e0d-b1c3-62d8213002df None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:07.909364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 375/736] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 15 13:57:07 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:07.910942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b6619f-8950-4cb3-b642-d06167697a18 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:07.911445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b6619f-8950-4cb3-b642-d06167697a18 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:07.911946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b6619f-8950-4cb3-b642-d06167697a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:07.918014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:07.932325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-26b6619f-8950-4cb3-b642-d06167697a18 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:07.933870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-26b6619f-8950-4cb3-b642-d06167697a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:07.978993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06efcd0d-85e7-4dd5-8a6b-209b52d28587 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.008885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 376/737] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:57:08.010139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b6619f-8950-4cb3-b642-d06167697a18 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:08.017014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562b4c68-18e7-400b-9929-82259648d869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.017772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562b4c68-18e7-400b-9929-82259648d869 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.018060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562b4c68-18e7-400b-9929-82259648d869 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.018336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562b4c68-18e7-400b-9929-82259648d869 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.092711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562b4c68-18e7-400b-9929-82259648d869 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.094265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562b4c68-18e7-400b-9929-82259648d869 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:08.098132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:08.117753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b6619f-8950-4cb3-b642-d06167697a18 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.149785 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:08.155101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac0944a-e811-4887-8f3b-58f3e9eb0c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.156016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0944a-e811-4887-8f3b-58f3e9eb0c25 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.156523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0944a-e811-4887-8f3b-58f3e9eb0c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.157005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0944a-e811-4887-8f3b-58f3e9eb0c25 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.209404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 362/738] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:57:08.214701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562b4c68-18e7-400b-9929-82259648d869 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:08.229658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac0944a-e811-4887-8f3b-58f3e9eb0c25 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.231487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac0944a-e811-4887-8f3b-58f3e9eb0c25 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:08.323451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562b4c68-18e7-400b-9929-82259648d869 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.328624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:08.356312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 377/739] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 15 13:57:08.363279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac0944a-e811-4887-8f3b-58f3e9eb0c25 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:08.367373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.371739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.371739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.371739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.380100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.381455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:08.390797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:08.400003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:08.400976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:08.401852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:08.401852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:08.401852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:08.401852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 363/740] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:08.407442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:08.494068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac0944a-e811-4887-8f3b-58f3e9eb0c25 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.526736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b504406f-2630-4d1f-a0d8-75f57917e1bb None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.537587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.538514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.538993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.539477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.550983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.553060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:08.565229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.565858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.566140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.566415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.567087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:08.567389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.568926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:08.571953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:08.572930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:08.573808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:08.573808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:08.573808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:08.576087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 378/741] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:08.577196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:08.589327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 364/742] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:08.590549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:08.631266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:08.670415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:08.702302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61a891-9b81-447b-bc60-22958e722f76 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.717917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db09aea-af9f-4b51-8f9a-5b7e471618f1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.738861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e880bc5f-1148-44e3-bcee-43e6691be6cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.739461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880bc5f-1148-44e3-bcee-43e6691be6cc None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.740176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880bc5f-1148-44e3-bcee-43e6691be6cc None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.740453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880bc5f-1148-44e3-bcee-43e6691be6cc None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.741100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e880bc5f-1148-44e3-bcee-43e6691be6cc None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.742456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e880bc5f-1148-44e3-bcee-43e6691be6cc None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:08.755347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e0cec1-5ae5-4fc2-af99-8b253a6b4e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.755940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e0cec1-5ae5-4fc2-af99-8b253a6b4e40 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.756274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e0cec1-5ae5-4fc2-af99-8b253a6b4e40 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.756552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e0cec1-5ae5-4fc2-af99-8b253a6b4e40 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.757314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e0cec1-5ae5-4fc2-af99-8b253a6b4e40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:08.757618 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e0cec1-5ae5-4fc2-af99-8b253a6b4e40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.758993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e0cec1-5ae5-4fc2-af99-8b253a6b4e40 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:08.768225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 379/743] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:57:08 2022] POST /identity/v3/services => generated 251 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:08.769682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e880bc5f-1148-44e3-bcee-43e6691be6cc None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:08.776121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 365/744] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:08.776121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feb3b00-254c-40b7-bada-bb220017fd4c None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.776121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feb3b00-254c-40b7-bada-bb220017fd4c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.776121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feb3b00-254c-40b7-bada-bb220017fd4c None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.776121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 366/745] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:08.777936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feb3b00-254c-40b7-bada-bb220017fd4c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:08.904911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:08.925149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e880bc5f-1148-44e3-bcee-43e6691be6cc None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.927120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feb3b00-254c-40b7-bada-bb220017fd4c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:08.950813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:08.976845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a267d5bc-7daa-4aae-bb7f-adb16e08a395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.977482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a267d5bc-7daa-4aae-bb7f-adb16e08a395 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.977765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a267d5bc-7daa-4aae-bb7f-adb16e08a395 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.978041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a267d5bc-7daa-4aae-bb7f-adb16e08a395 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.984953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:08.985599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:08.985877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:08.986150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:08.986735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a267d5bc-7daa-4aae-bb7f-adb16e08a395 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.989067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a267d5bc-7daa-4aae-bb7f-adb16e08a395 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:08.993995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:08.995420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a267d5bc-7daa-4aae-bb7f-adb16e08a395 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:09.000979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:09.002031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 380/746] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:09.003443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2338c6d9-b796-4cb6-a5a3-9a3e5036262d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.004432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2338c6d9-b796-4cb6-a5a3-9a3e5036262d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.004432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2338c6d9-b796-4cb6-a5a3-9a3e5036262d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:09.012945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:09.013847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:09.014751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 367/747] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:09.015246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:09.027013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2338c6d9-b796-4cb6-a5a3-9a3e5036262d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:09.028242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2338c6d9-b796-4cb6-a5a3-9a3e5036262d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:09.129205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 381/748] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:09 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:57:09.135767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2338c6d9-b796-4cb6-a5a3-9a3e5036262d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:09.136285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b7abf4-49e0-4568-af7f-114e8a1c7f06 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:09.176533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:09.177531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.178572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.179047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.179962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:09.180464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:09.182367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:09.197480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 368/749] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:09.203610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:09.242841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:09.322508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f699a7-953a-4599-9a6c-e7031882a104 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:09.345835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:09.363816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:09.364654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.365079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.365493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.366388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=75c8381b6ad84cac881db621d2d99af7) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:09.366840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:09.368435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:09.375524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 369/750] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:09 2022] GET /identity/v3/role_assignments?user.id=75c8381b6ad84cac881db621d2d99af7&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:09.377492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2338c6d9-b796-4cb6-a5a3-9a3e5036262d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:09.419031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ec4f79-c206-4a77-9300-a9cd7155a87c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:09.420362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ec4f79-c206-4a77-9300-a9cd7155a87c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.420849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ec4f79-c206-4a77-9300-a9cd7155a87c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.421299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ec4f79-c206-4a77-9300-a9cd7155a87c None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.422245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ec4f79-c206-4a77-9300-a9cd7155a87c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:09.422786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ec4f79-c206-4a77-9300-a9cd7155a87c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:09.424714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ec4f79-c206-4a77-9300-a9cd7155a87c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:09.440586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 382/751] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:09.448862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:09.545218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf30a701-7c67-4eb5-a6d9-2bb7d8b991cc None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.546196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf30a701-7c67-4eb5-a6d9-2bb7d8b991cc None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.546975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf30a701-7c67-4eb5-a6d9-2bb7d8b991cc None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.549464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 383/752] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:09.550962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf30a701-7c67-4eb5-a6d9-2bb7d8b991cc None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:09.568718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:09.572485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586fac30-8cf4-407f-957a-40798d4bf74e None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:09.612805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd3716b-2083-4ade-88ca-a9513294bfb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:09.613417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd3716b-2083-4ade-88ca-a9513294bfb1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.613699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd3716b-2083-4ade-88ca-a9513294bfb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.614044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd3716b-2083-4ade-88ca-a9513294bfb1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.614774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd3716b-2083-4ade-88ca-a9513294bfb1 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=30fc2a26b4fc4d24b984fa8097752055) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:09.615084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd3716b-2083-4ade-88ca-a9513294bfb1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:09.616551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd3716b-2083-4ade-88ca-a9513294bfb1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:09.624884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 370/753] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:09 2022] GET /identity/v3/role_assignments?user.id=30fc2a26b4fc4d24b984fa8097752055&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:09.626522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b365d55-5965-4086-b329-51bbbd018603 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.626812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b365d55-5965-4086-b329-51bbbd018603 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.627704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b365d55-5965-4086-b329-51bbbd018603 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.649013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-8b365d55-5965-4086-b329-51bbbd018603 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:09.650249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8b365d55-5965-4086-b329-51bbbd018603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:09.699419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf30a701-7c67-4eb5-a6d9-2bb7d8b991cc None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:09.742829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0248b138-da30-45b7-b4a4-cbeb5af99366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:09.743893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248b138-da30-45b7-b4a4-cbeb5af99366 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.744644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248b138-da30-45b7-b4a4-cbeb5af99366 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.745125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248b138-da30-45b7-b4a4-cbeb5af99366 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.760002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0248b138-da30-45b7-b4a4-cbeb5af99366 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:09.760224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 371/754] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:09 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-0248b138-da30-45b7-b4a4-cbeb5af99366 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:09.765475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:09.766458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:09.767304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:09.770209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 384/755] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:09.774567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:09.776065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b365d55-5965-4086-b329-51bbbd018603 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:09.931569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:09.939720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b365d55-5965-4086-b329-51bbbd018603 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:09.975974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:09.976652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:09.977047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:09.977394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:09.978119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:09.978428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:09.979836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:09.989749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 372/756] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:09.995188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0248b138-da30-45b7-b4a4-cbeb5af99366 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:10.084193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:10.116334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0248b138-da30-45b7-b4a4-cbeb5af99366 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:10.179981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:10.180690 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:10.180987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:10.181245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:10.202210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:10.203579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:10.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:10.231393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:10.232661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:10.232661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:10.232661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:10.234730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 385/757] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:10.243773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:10.265078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b5d6b4-fab5-4eb0-beef-913c8a54abe6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:10.335967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5187d8-1229-4dcd-b956-a878d7fd1164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:10.336580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5187d8-1229-4dcd-b956-a878d7fd1164 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:10.336858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5187d8-1229-4dcd-b956-a878d7fd1164 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:10.337135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5187d8-1229-4dcd-b956-a878d7fd1164 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:10.337803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5187d8-1229-4dcd-b956-a878d7fd1164 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:10.339158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5187d8-1229-4dcd-b956-a878d7fd1164 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:10.370765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 373/758] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:57:10 2022] POST /identity/v3/services => generated 240 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:10.372659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:10.454083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b56d895-7d6c-4af8-91e1-2414ad55442b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:10.508958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:10.509575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:10.509852 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:10.510127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:10.510779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:10.511082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:10.512572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:10.532034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 386/759] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:10.533703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5187d8-1229-4dcd-b956-a878d7fd1164 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:10.560054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:10.640928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5187d8-1229-4dcd-b956-a878d7fd1164 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:10.681626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:10.682315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:10.682602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:10.682873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:10.691353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:10.692876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:10.705546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:10.706501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:10.707905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 374/760] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:57:10 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:10.712915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:10.822741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf380ff-733b-4974-bab0-ae9ad68b4b67 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:10.833588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:10.865694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:10.866324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:10.866603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:10.866872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:10.867614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:10.867953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:10.869396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:10.889144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 387/761] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:57:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:10.896266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:11.010143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:11.032993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06899ca-ef1e-459d-8aca-56ed7b57a054 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:11.067433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:11.068087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:11.068362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:11.068629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:11.078974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:11.080623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:11.094726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:11.096367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:11.097871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:11.097871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:11.097871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:11.097871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 375/762] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:11.104813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:11.250447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f3d563-6062-43a7-92b5-9d4dcfda3b6b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:11.299748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:11.300743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:11.301471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:11.301944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:11.302802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:11.303304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:11.304949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:11.325109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 388/763] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:11.331466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:11.482801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79b7d27-1741-4528-aee6-d55852ec18ed None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:11.524701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:11.532945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff7240a-0610-4042-ad3f-7d5ba7525b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:11.533660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff7240a-0610-4042-ad3f-7d5ba7525b27 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:11.534019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff7240a-0610-4042-ad3f-7d5ba7525b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:11.534209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff7240a-0610-4042-ad3f-7d5ba7525b27 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:11.544914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff7240a-0610-4042-ad3f-7d5ba7525b27 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:11.546414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff7240a-0610-4042-ad3f-7d5ba7525b27 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-dff7240a-0610-4042-ad3f-7d5ba7525b27 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:11.560144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:11.561014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:11.562228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 376/764] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:11.567537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:11.573323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a04a9b-2717-4b21-b703-a6bde6e75f89 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:11.573836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a04a9b-2717-4b21-b703-a6bde6e75f89 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:11.574523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a04a9b-2717-4b21-b703-a6bde6e75f89 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:11.577392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 377/765] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:11.583993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19519bf-03b4-4013-a4de-cd30f41d914a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:11.584514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19519bf-03b4-4013-a4de-cd30f41d914a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:11.584970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19519bf-03b4-4013-a4de-cd30f41d914a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:11.617050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d19519bf-03b4-4013-a4de-cd30f41d914a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:11.618244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d19519bf-03b4-4013-a4de-cd30f41d914a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:11.626869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:11.732768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 378/766] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:11 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:11.767912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05919f53-c0d9-4734-8920-398585f4bda4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:11.805549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:11.806184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:11.806463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:11.806741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:11.807430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:11.807762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:11.809192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:11.822601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 389/767] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:11.828315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19519bf-03b4-4013-a4de-cd30f41d914a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:11.978582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19519bf-03b4-4013-a4de-cd30f41d914a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:12.001917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:12.012702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:12.025683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e6614a-753b-45b2-9718-82497ac2970c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:12.026352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6614a-753b-45b2-9718-82497ac2970c None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.026631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6614a-753b-45b2-9718-82497ac2970c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.026904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6614a-753b-45b2-9718-82497ac2970c None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/ce6744790db74f0aaffc6eb269bf6737/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.030054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e6614a-753b-45b2-9718-82497ac2970c None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=ce6744790db74f0aaffc6eb269bf6737, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:12.031487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e6614a-753b-45b2-9718-82497ac2970c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:12.052204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 379/768] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:57:11 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/ce6744790db74f0aaffc6eb269bf6737/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:57:12.053583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2d8fe-5ffb-4aff-99c4-88330336c199 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.053873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2d8fe-5ffb-4aff-99c4-88330336c199 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.054146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2d8fe-5ffb-4aff-99c4-88330336c199 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.055756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 380/769] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:12.061891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146532fd-5770-40d8-b66d-956f8a65bccc None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.062187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146532fd-5770-40d8-b66d-956f8a65bccc None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.062435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146532fd-5770-40d8-b66d-956f8a65bccc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.093796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-146532fd-5770-40d8-b66d-956f8a65bccc None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:12.098051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-146532fd-5770-40d8-b66d-956f8a65bccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:12.168508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 381/770] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:12.181565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:12.203648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946a5fa3-1e5a-48c6-ac59-7e74fd773cbd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:12.250681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:12.251321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.251600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.251955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.252660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:12.254030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:12.266085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:12.267118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:12.268547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 390/771] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Sep 15 13:57:12 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:12.274004 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146532fd-5770-40d8-b66d-956f8a65bccc None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:12.398055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146532fd-5770-40d8-b66d-956f8a65bccc None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:12.408801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:12.445040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:12.445666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.445941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.446210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.446856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:12.447153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:12.448558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:12.454172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 382/772] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 15 13:57:12 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:12.459893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:12.490717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:12.590834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9368983-16fb-4fab-bd79-f01a299135c2 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:12.615521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a150a97-dd3c-4226-96fb-b1012ea16ae5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:12.628049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854d5ca2-bf78-44d0-8596-85b245755fbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:12.628864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d5ca2-bf78-44d0-8596-85b245755fbf None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.629143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d5ca2-bf78-44d0-8596-85b245755fbf None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.629409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d5ca2-bf78-44d0-8596-85b245755fbf None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.630042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854d5ca2-bf78-44d0-8596-85b245755fbf None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:12.630340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854d5ca2-bf78-44d0-8596-85b245755fbf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:12.631737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854d5ca2-bf78-44d0-8596-85b245755fbf None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:12.637264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 391/773] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 15 13:57:12 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:12.638463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda23776-6310-4848-946c-935e8bbf03c9 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.638746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda23776-6310-4848-946c-935e8bbf03c9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.639018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda23776-6310-4848-946c-935e8bbf03c9 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.640496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 392/774] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:12.642785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda23776-6310-4848-946c-935e8bbf03c9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:12.654635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d2a818-2230-4ff6-aa11-ec3f5994a065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:12.655197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d2a818-2230-4ff6-aa11-ec3f5994a065 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.655472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d2a818-2230-4ff6-aa11-ec3f5994a065 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.655764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d2a818-2230-4ff6-aa11-ec3f5994a065 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.660975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:12.670365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d2a818-2230-4ff6-aa11-ec3f5994a065 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-77d2a818-2230-4ff6-aa11-ec3f5994a065 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:12.676654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:12.677598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:12.678450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 383/775] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:57:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:12.680587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391dbce8-7796-411e-919f-b5528b9b9e8a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.680587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391dbce8-7796-411e-919f-b5528b9b9e8a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.681088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391dbce8-7796-411e-919f-b5528b9b9e8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.706784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-391dbce8-7796-411e-919f-b5528b9b9e8a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:12.708789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-391dbce8-7796-411e-919f-b5528b9b9e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:12.746624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:12.767372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda23776-6310-4848-946c-935e8bbf03c9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:12.804713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 384/776] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:12.804713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391dbce8-7796-411e-919f-b5528b9b9e8a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:12.817787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:12.818435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:12.818684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:12.818958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:12.819673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] RBAC: Adding query filter params (interface=public, service_id=93c721eb56514994903da364ba8a78a7, region_id=RegionOne) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:12.820004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:12.821394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:12.841885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 393/777] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 15 13:57:12 2022] GET /identity/v3/endpoints?service_id=93c721eb56514994903da364ba8a78a7&interface=public®ion_id=RegionOne => generated 194 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:12.847555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:12.960161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391dbce8-7796-411e-919f-b5528b9b9e8a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:12.990555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:13.021387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:13.021987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.022267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.022541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.023217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:13.023517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:13.024911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:13.034001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d29465b-142c-4704-aa1c-fe0779a7d699 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:13.037893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 385/778] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:57:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:13.039600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:13.069624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:13.128453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2da94-7f6a-4d72-b503-4928b73462dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:13.129076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2da94-7f6a-4d72-b503-4928b73462dd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.129356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2da94-7f6a-4d72-b503-4928b73462dd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.129628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2da94-7f6a-4d72-b503-4928b73462dd None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.130236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e2da94-7f6a-4d72-b503-4928b73462dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:13.131612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e2da94-7f6a-4d72-b503-4928b73462dd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:13.140750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 394/779] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 13:57:12 2022] GET /identity/v3/services => generated 772 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:13.141973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2da94-7f6a-4d72-b503-4928b73462dd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:13.173628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bec614-b032-4d76-8b51-ce9cf433e8ad None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:13.264685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:13.265706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.266174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.266641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.305908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:13.307405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:13.338603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:13.339629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:13.340552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:13.340552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:13.340552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:13.342957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 386/780] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:13.353993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:13.362587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2da94-7f6a-4d72-b503-4928b73462dd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:13.407125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e983cd-a055-4e8b-a2ea-5587d4f977f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:13.407942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e983cd-a055-4e8b-a2ea-5587d4f977f1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.408870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e983cd-a055-4e8b-a2ea-5587d4f977f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.409633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e983cd-a055-4e8b-a2ea-5587d4f977f1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.422308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e983cd-a055-4e8b-a2ea-5587d4f977f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-29e983cd-a055-4e8b-a2ea-5587d4f977f1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:13.428549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:13.429528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:13.430371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 395/781] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:13.435573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d22bc-4278-4955-b18b-4715c325f52f None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.435908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d22bc-4278-4955-b18b-4715c325f52f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.436189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d22bc-4278-4955-b18b-4715c325f52f None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.437476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 396/782] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:13 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:13.442129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d22bc-4278-4955-b18b-4715c325f52f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:13.443801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:13.453422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:13.482802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88a1c50-e669-49c3-a2eb-e64811d049f5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:13.520683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fcd54d-4bdd-451a-9ad6-6ce62f95c4bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:13.521661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fcd54d-4bdd-451a-9ad6-6ce62f95c4bc None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.522143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fcd54d-4bdd-451a-9ad6-6ce62f95c4bc None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.522612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fcd54d-4bdd-451a-9ad6-6ce62f95c4bc None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.523453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fcd54d-4bdd-451a-9ad6-6ce62f95c4bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:13.523985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fcd54d-4bdd-451a-9ad6-6ce62f95c4bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:13.525621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fcd54d-4bdd-451a-9ad6-6ce62f95c4bc None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:13.538597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 387/783] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:13.540242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23afe2-28ab-48c2-bfc7-0ecd1359d7e3 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.540744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23afe2-28ab-48c2-bfc7-0ecd1359d7e3 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.541212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23afe2-28ab-48c2-bfc7-0ecd1359d7e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.560587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-3b23afe2-28ab-48c2-bfc7-0ecd1359d7e3 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:13.562172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3b23afe2-28ab-48c2-bfc7-0ecd1359d7e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:13.569257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d22bc-4278-4955-b18b-4715c325f52f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:13.606322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:13.606965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.607239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.607509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.608216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:13.608544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:13.610029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:13.623871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 397/784] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:13.624759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:13.636619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 388/785] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:13 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:13.644788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23afe2-28ab-48c2-bfc7-0ecd1359d7e3 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:13.694874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:13.735913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0409552-2eaa-4aee-882a-4ded74e0d74a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:13.744796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:13.773360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:13.773913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.774197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.774466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.779326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23afe2-28ab-48c2-bfc7-0ecd1359d7e3 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:13.782490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:13.785076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:13.797261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:13.798259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:13.799874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 398/786] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 13:57:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:13.804473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:13.817153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:13.817887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:13.818174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:13.818455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:13.829280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:13.831538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:13.847517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:13.848493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:13.849960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:13.849960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:13.849960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:13.849960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 389/787] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:13.859851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:13.950031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d3e6a0-84dc-45d3-b606-37ca683c1cb3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.000698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:14.025079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.025612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.025888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.026157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.026892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:14.027192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.031167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.049987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ba268c-7bf4-4e14-b750-2b282b16a5ca None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.074022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 399/788] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 15 13:57:13 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:14.074983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.085247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:14.106847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.107455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.107765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.108039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.108680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:14.108977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.111089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.124716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 390/789] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:14.128616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.197243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f67eb-03f7-47b5-a9d4-68e41b991155 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.234997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.235707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.236010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.236854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] PATH_INFO: `/v3/services/network` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.237543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.239048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.241084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d42e7b3-eaa3-4cca-8d21-d805c45e20b7 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:14.248854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:14.249843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:14.250675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:14.255987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 400/790] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/services/network => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:14.256882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.278841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.279393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.279668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.279961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.291757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.293185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:14.310076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:14.312347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:14.314653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:14.314653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:14.314653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:14.324231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 391/791] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:14.324231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.324582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:14.351565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:14.385653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7b3faf-76e6-4ac0-b849-ee1f152a0ba1 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.423186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.424585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.424873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.425164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.428663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75becc98-eba5-4e53-9140-75c3d8b4c33d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.433902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.435354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:14.447717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:14.450005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 401/792] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/users/swift => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:14.452279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.470553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.471207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.471449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.471756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.472389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] RBAC: Adding query filter params (name=network) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:14.472689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.474040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.479002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 392/793] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/services?name=network => generated 124 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:14.479908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.576300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e623fdfa-32d1-41b2-8086-a7a6df83f994 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.584708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:14.597341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:14.605076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2918ca-7947-49d6-8bf4-1f6e0397b40c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.619765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.620711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.621144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.621568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.622363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:14.622929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.624571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.638983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 402/794] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:14.640128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.648790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.649616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.649898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.650166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.650899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:14.651201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.652647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.671512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 393/795] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:14.676846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.750785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e3e99-3630-4cc1-8863-1eb9d5a74851 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.781555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435e1987-596f-4f46-86b5-782e26f90d65 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.787778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8496d701-5a3e-4279-8228-26ca0d0cf6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.788384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8496d701-5a3e-4279-8228-26ca0d0cf6f2 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.788660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8496d701-5a3e-4279-8228-26ca0d0cf6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.789268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8496d701-5a3e-4279-8228-26ca0d0cf6f2 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.789619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8496d701-5a3e-4279-8228-26ca0d0cf6f2 None admin] RBAC: Adding query filter params (type=network) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:14.789917 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8496d701-5a3e-4279-8228-26ca0d0cf6f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.791280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8496d701-5a3e-4279-8228-26ca0d0cf6f2 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.796420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 403/796] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/services?type=network => generated 350 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:14.797546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92173cc8-5379-4341-a6ff-d7706df5bcff None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.797826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92173cc8-5379-4341-a6ff-d7706df5bcff None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.798092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92173cc8-5379-4341-a6ff-d7706df5bcff None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.803589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 404/797] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:14 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:14.803589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92173cc8-5379-4341-a6ff-d7706df5bcff None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.828684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.829244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.829527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.829800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.836666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:14.837257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.839023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:14.853266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:14.854272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 394/798] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:14.860052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:14.878397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:14.925785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92173cc8-5379-4341-a6ff-d7706df5bcff None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.967599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10ad0e1-3343-4746-8552-d9545efb2ae1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:14.968518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce5e09c-63a3-4db9-bb15-87f156fc915d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:14.969265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5e09c-63a3-4db9-bb15-87f156fc915d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:14.969618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5e09c-63a3-4db9-bb15-87f156fc915d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:14.969968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5e09c-63a3-4db9-bb15-87f156fc915d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:14.981292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce5e09c-63a3-4db9-bb15-87f156fc915d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:14.982738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce5e09c-63a3-4db9-bb15-87f156fc915d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-4ce5e09c-63a3-4db9-bb15-87f156fc915d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:14.998314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:14.999265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:15.000230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:15.000230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:15.000230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:15.000230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 405/799] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:15.001012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669b320-582d-40fe-9e75-d8216a782103 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.001012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669b320-582d-40fe-9e75-d8216a782103 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.001186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669b320-582d-40fe-9e75-d8216a782103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.013680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:15.014382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.014735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.015085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.015921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] RBAC: Adding query filter params (interface=public, service_id=75b089c53bfe438a9d75e19e3aae0cd9, region_id=RegionOne) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:15.016234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:15.018161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:15.020323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8669b320-582d-40fe-9e75-d8216a782103 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:15.021514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8669b320-582d-40fe-9e75-d8216a782103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:15.031144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 395/800] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 15 13:57:14 2022] GET /identity/v3/endpoints?service_id=75b089c53bfe438a9d75e19e3aae0cd9&interface=public®ion_id=RegionOne => generated 194 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:15.031144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.102376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 406/801] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:15.103433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8669b320-582d-40fe-9e75-d8216a782103 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.121440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:15.126420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:15.150313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a4ddbf-0156-4c98-84e2-de06a7c38426 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:15.188335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:15.189101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.190146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.190519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.191442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:15.191982 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:15.193504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:15.206487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 396/802] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:15.207553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.239586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8669b320-582d-40fe-9e75-d8216a782103 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:15.291110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:15.291851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.292164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.292443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.293512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:15.293830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:15.295574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:15.325952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 407/803] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:15.325952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.334291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5863b15a-cc6d-4223-8797-c59e36df674c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:15.378172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85b278-7910-423d-9b91-597a39b5f390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:15.378715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85b278-7910-423d-9b91-597a39b5f390 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.378996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85b278-7910-423d-9b91-597a39b5f390 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.379275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85b278-7910-423d-9b91-597a39b5f390 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.380508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e85b278-7910-423d-9b91-597a39b5f390 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:15.382540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e85b278-7910-423d-9b91-597a39b5f390 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:15.385475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:15.389974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 397/804] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 13:57:15 2022] GET /identity/v3/services => generated 772 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:15.391017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85b278-7910-423d-9b91-597a39b5f390 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.424652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:15.472333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9854ef00-1f8b-4b4f-972c-b428e7b090bd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:15.518901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b92c402-e255-4fbb-a7c7-91378765b48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:15.519586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b92c402-e255-4fbb-a7c7-91378765b48b None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.519891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b92c402-e255-4fbb-a7c7-91378765b48b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.520175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b92c402-e255-4fbb-a7c7-91378765b48b None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/75c8381b6ad84cac881db621d2d99af7/roles/91265edc38fd488492ee9ae7ab7d2440` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.525287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85b278-7910-423d-9b91-597a39b5f390 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:15.537300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b92c402-e255-4fbb-a7c7-91378765b48b None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=75c8381b6ad84cac881db621d2d99af7, role_id=91265edc38fd488492ee9ae7ab7d2440)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:15.538772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b92c402-e255-4fbb-a7c7-91378765b48b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:15.556004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 408/805] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:57:15 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/75c8381b6ad84cac881db621d2d99af7/roles/91265edc38fd488492ee9ae7ab7d2440 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:57:15.556887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b92c402-e255-4fbb-a7c7-91378765b48b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.585446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c1c512-4db3-4dff-888e-56860911be8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:15.586667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1c512-4db3-4dff-888e-56860911be8b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.587194 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1c512-4db3-4dff-888e-56860911be8b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.587724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1c512-4db3-4dff-888e-56860911be8b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.598285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c1c512-4db3-4dff-888e-56860911be8b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:15.600201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c1c512-4db3-4dff-888e-56860911be8b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-75c1c512-4db3-4dff-888e-56860911be8b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:15.615280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:15.616311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:15.617206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:15.617206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:15.617206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:15.617206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:15.617206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:15.617206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:15.617206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:15.617206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:15.617654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 398/806] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:15.618897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80c7f3b-c944-45cd-947d-8014f23981cb None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.619022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80c7f3b-c944-45cd-947d-8014f23981cb None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.619899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80c7f3b-c944-45cd-947d-8014f23981cb None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.623734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 399/807] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:15.624362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80c7f3b-c944-45cd-947d-8014f23981cb None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.697576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b92c402-e255-4fbb-a7c7-91378765b48b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:15.707619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:15.729734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:15.746926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eb54f4-15c5-4e82-8e6c-249c542e8c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:15.747568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eb54f4-15c5-4e82-8e6c-249c542e8c72 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.747931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eb54f4-15c5-4e82-8e6c-249c542e8c72 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.748211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eb54f4-15c5-4e82-8e6c-249c542e8c72 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.760100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eb54f4-15c5-4e82-8e6c-249c542e8c72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:15.761696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80c7f3b-c944-45cd-947d-8014f23981cb None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-b0eb54f4-15c5-4e82-8e6c-249c542e8c72 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:15.765525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:15.766489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:15.767930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 409/808] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:15.768724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f1ad5d-946e-4144-ad51-d183d33ad8df None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.768724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f1ad5d-946e-4144-ad51-d183d33ad8df None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.768858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f1ad5d-946e-4144-ad51-d183d33ad8df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.794606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-82f1ad5d-946e-4144-ad51-d183d33ad8df None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:15.795860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-82f1ad5d-946e-4144-ad51-d183d33ad8df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:15.812917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:15.813678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:15.814056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:15.814515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:15.815327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:15.815758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:15.817269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:15.832999 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 400/809] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:15.838264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.887263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 410/810] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:15.888681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f1ad5d-946e-4144-ad51-d183d33ad8df None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:15.956474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c22b94-8963-4177-a106-ce5a159ed19d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:15.979553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:16.003590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:16.004470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:16.008088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:16.008530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:16.009364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:16.009837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:16.011359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:16.029717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 401/811] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:16.036084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:16.038123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f1ad5d-946e-4144-ad51-d183d33ad8df None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:16.068063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:16.109216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adb8a4-bcd1-4836-99e6-dd1f79e16b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:16.109882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adb8a4-bcd1-4836-99e6-dd1f79e16b3b None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:16.110162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adb8a4-bcd1-4836-99e6-dd1f79e16b3b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:16.110435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adb8a4-bcd1-4836-99e6-dd1f79e16b3b None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/30fc2a26b4fc4d24b984fa8097752055/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:16.118912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50adb8a4-bcd1-4836-99e6-dd1f79e16b3b None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=30fc2a26b4fc4d24b984fa8097752055, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:16.120552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50adb8a4-bcd1-4836-99e6-dd1f79e16b3b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:16.142713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 411/812] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:57:15 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/30fc2a26b4fc4d24b984fa8097752055/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:57:16.221751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154100c6-d8f1-46cf-97b3-809a7c4e4ab1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:16.240832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adb8a4-bcd1-4836-99e6-dd1f79e16b3b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:16.271717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:16.272364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:16.272644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:16.272976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:16.288098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:16.289557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:16.317189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:16.318120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:16.319024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:16.319024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:16.319024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:16.321316 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 402/813] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:16.327745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:16.396409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:16.485244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adb8a4-bcd1-4836-99e6-dd1f79e16b3b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:16.508474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5e3059-85b3-452c-87a2-9604178126fd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:16.524037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:16.524680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:16.524962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:16.525237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:16.525932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:16.527430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:16.537094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:16.538110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:16.538964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 412/814] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Sep 15 13:57:16 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:16.544252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:16.545314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:16.545923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:16.546201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:16.546470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:16.547114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:16.547413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:16.549374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:16.562984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 403/815] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:16.569647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:16.591283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:16.670739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a164e5d1-19c0-45b2-924d-8d94b3e804ca None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:16.732431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:16.741262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:16.741498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65de30a-4bdb-4dfe-bb99-e3394ca7aff0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:16.741816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:16.742096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:16.742369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:16.743178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:16.743485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:16.745681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:16.758424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 413/816] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 15 13:57:16 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:16.764509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:16.807519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:16.808544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:16.809055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:16.809530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] PATH_INFO: `/v3/users/nova` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:16.823533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:16.825325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:16.849753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:16.850753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:16.851720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:16.854329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 404/817] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:57:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:16.860130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:16.877203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e72905-7c88-4169-b8eb-1f5c6e24c1e3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:16.878811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:16.915714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:16.916331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:16.916612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:16.916898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:16.917820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:16.918238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:16.920043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:16.926713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 414/818] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 15 13:57:16 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:16.932237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:17.039354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99030f6c-e9e6-42f5-a75b-9c7a9da5a322 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:17.064922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4e6f8-2abc-4346-8bf0-b1fb044ac9df None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:17.075605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b790f9d-ccbe-40d8-a1b9-75dfe78584e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:17.076359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b790f9d-ccbe-40d8-a1b9-75dfe78584e7 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:17.076711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b790f9d-ccbe-40d8-a1b9-75dfe78584e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:17.077056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b790f9d-ccbe-40d8-a1b9-75dfe78584e7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:17.077794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b790f9d-ccbe-40d8-a1b9-75dfe78584e7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:17.079736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b790f9d-ccbe-40d8-a1b9-75dfe78584e7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:17.104914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 415/819] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 13:57:16 2022] POST /identity/v3/endpoints => generated 346 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:17.156556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:17.157516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:17.158510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:17.159164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:17.160195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:17.160703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:17.162381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:17.188326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 405/820] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:57:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:17.194747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b790f9d-ccbe-40d8-a1b9-75dfe78584e7 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:17.225790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:17.434787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b790f9d-ccbe-40d8-a1b9-75dfe78584e7 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:17.471894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d539fba3-142e-425d-bd6c-3b1b93bb518a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:17.472516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d539fba3-142e-425d-bd6c-3b1b93bb518a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:17.472831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d539fba3-142e-425d-bd6c-3b1b93bb518a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:17.473105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d539fba3-142e-425d-bd6c-3b1b93bb518a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:17.545201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d539fba3-142e-425d-bd6c-3b1b93bb518a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:17.546676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d539fba3-142e-425d-bd6c-3b1b93bb518a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:17.571419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:17.667091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 416/821] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:17 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:57:17.672283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:17.684304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:17.801732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fa2d5a-45c9-4a52-be60-3f1fe8068f55 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:17.858287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e63905a-dcfb-41d2-b2de-c99e7bbd665d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:17.859254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e63905a-dcfb-41d2-b2de-c99e7bbd665d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:17.859736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e63905a-dcfb-41d2-b2de-c99e7bbd665d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:17.860278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e63905a-dcfb-41d2-b2de-c99e7bbd665d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:17.872805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e63905a-dcfb-41d2-b2de-c99e7bbd665d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:17.874567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e63905a-dcfb-41d2-b2de-c99e7bbd665d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-9e63905a-dcfb-41d2-b2de-c99e7bbd665d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:17.896321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:17.897330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:17.898227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:17.898227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:17.898227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:17.899793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 406/822] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:17.905449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d539fba3-142e-425d-bd6c-3b1b93bb518a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:17.980588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29bd2dc-35c8-4cc6-ac53-45b3fdc7031a None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:17.981383 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29bd2dc-35c8-4cc6-ac53-45b3fdc7031a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:17.981883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29bd2dc-35c8-4cc6-ac53-45b3fdc7031a None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:17.984575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 407/823] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:17.985912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73567c6e-dab6-4ad9-931c-dbca6dff1ba9 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:17.986611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73567c6e-dab6-4ad9-931c-dbca6dff1ba9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:17.986951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73567c6e-dab6-4ad9-931c-dbca6dff1ba9 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:17.988419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 408/824] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:17.989458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d22827-67bb-4f5f-bdd2-16a180bd5f15 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:17.989732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d22827-67bb-4f5f-bdd2-16a180bd5f15 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:17.989998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d22827-67bb-4f5f-bdd2-16a180bd5f15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.012876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-79d22827-67bb-4f5f-bdd2-16a180bd5f15 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:18.014063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-79d22827-67bb-4f5f-bdd2-16a180bd5f15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:18.028008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d539fba3-142e-425d-bd6c-3b1b93bb518a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:18.064110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:18.076137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:18.077630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.078124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.078670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.079573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:18.080169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:18.082223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:18.084783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:18.098282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 409/825] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:18.098800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 417/826] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:18.099535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f852e74-da30-4f33-b4db-95976e65b7e1 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.099846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f852e74-da30-4f33-b4db-95976e65b7e1 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.100116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f852e74-da30-4f33-b4db-95976e65b7e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.103601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:18.131591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2f852e74-da30-4f33-b4db-95976e65b7e1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:18.134773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2f852e74-da30-4f33-b4db-95976e65b7e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:18.256374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 410/827] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:18 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:18.303972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e361eb-23c5-4e51-bac8-afc116fb35f7 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:18.366237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:18.366879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.367161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.367439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.373214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f852e74-da30-4f33-b4db-95976e65b7e1 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:18.377403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:18.379318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:18.390834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:18.391941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:18.393028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:18.393028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:18.393028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:18.393028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:18.393028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:18.393028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:18.393028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:18.393028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:18.395243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 418/828] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:18.400472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:18.468389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:18.582193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f852e74-da30-4f33-b4db-95976e65b7e1 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:18.584573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e1b2c5-cbe8-44f3-b7d7-47db737a1370 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:18.611395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:18.629342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d374aa7-5cb6-46af-afb5-dd489d2f9755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:18.630137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d374aa7-5cb6-46af-afb5-dd489d2f9755 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.630434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d374aa7-5cb6-46af-afb5-dd489d2f9755 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.630773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d374aa7-5cb6-46af-afb5-dd489d2f9755 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.632556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:18.633371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.633850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.634271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.635098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:18.635544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:18.637118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:18.641784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d374aa7-5cb6-46af-afb5-dd489d2f9755 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-7d374aa7-5cb6-46af-afb5-dd489d2f9755 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:18.646678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:18.647824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:18.648697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 411/829] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:57:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:18.649459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d374aa7-5cb6-46af-afb5-dd489d2f9755 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:18.654895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 419/830] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:18.655949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:18.766432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:18.769836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315de831-7a66-41da-83b1-25aa9902ffb6 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:18.770492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d374aa7-5cb6-46af-afb5-dd489d2f9755 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:18.809237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:18.810141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.810620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.810823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703220de-8806-4de0-a235-d2694c407996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:18.811254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.811458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703220de-8806-4de0-a235-d2694c407996 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.811762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703220de-8806-4de0-a235-d2694c407996 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.812045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703220de-8806-4de0-a235-d2694c407996 None admin] PATH_INFO: `/v3/services/network` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.812664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:18.812766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703220de-8806-4de0-a235-d2694c407996 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:18.813236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:18.814045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703220de-8806-4de0-a235-d2694c407996 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:18.815357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:18.824412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 420/831] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:57:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-703220de-8806-4de0-a235-d2694c407996 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:18.825486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:18.826450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:18.827287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 412/832] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:18 2022] GET /identity/v3/services/network => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:18.828111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:18.834009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703220de-8806-4de0-a235-d2694c407996 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:18.882912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:18.941651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f9b077-706c-4797-a2b4-3f1806c9a9c8 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:18.943984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703220de-8806-4de0-a235-d2694c407996 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:18.981556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:18.982198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.982491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.982773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.983629 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=ce6744790db74f0aaffc6eb269bf6737) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:18.984039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:18.986041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:18.987399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:18.988396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:18.988874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:18.989342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:18.994347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 421/833] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:18 2022] GET /identity/v3/role_assignments?user.id=ce6744790db74f0aaffc6eb269bf6737&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:18.995234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:19.004720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:19.006601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:19.023998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:19.025017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:19.025891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:19.025891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:19.025891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:19.028141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 413/834] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:19.039718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:19.046506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:19.130498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62543d98-0284-444c-8b72-bbe32e01e911 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:19.170164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff50b4c0-f70d-4ed3-ad79-a60a7c597d70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:19.170798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff50b4c0-f70d-4ed3-ad79-a60a7c597d70 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.171080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff50b4c0-f70d-4ed3-ad79-a60a7c597d70 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.171359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff50b4c0-f70d-4ed3-ad79-a60a7c597d70 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.172042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff50b4c0-f70d-4ed3-ad79-a60a7c597d70 None admin] RBAC: Adding query filter params (name=network) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:19.172348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff50b4c0-f70d-4ed3-ad79-a60a7c597d70 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:19.173813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff50b4c0-f70d-4ed3-ad79-a60a7c597d70 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:19.180130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 422/835] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:18 2022] GET /identity/v3/services?name=network => generated 124 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:19.181413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23841d-1933-409b-a789-68d564ddc221 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.181704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23841d-1933-409b-a789-68d564ddc221 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.181982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23841d-1933-409b-a789-68d564ddc221 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.183421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 423/836] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:19.189009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c868f-6cbb-4080-ac37-108ba4c3bfff None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.189878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c868f-6cbb-4080-ac37-108ba4c3bfff None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.190154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c868f-6cbb-4080-ac37-108ba4c3bfff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.204100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe002d3-bd45-406c-adb3-4442e2ee7d3e None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:19.208421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:19.209954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8a5c868f-6cbb-4080-ac37-108ba4c3bfff None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:19.211293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8a5c868f-6cbb-4080-ac37-108ba4c3bfff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:19.253904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:19.254520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.254803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.255076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.255760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:19.256077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:19.257440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:19.273347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 414/837] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:19.274025 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:19.319297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 424/838] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:19.320294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5c868f-6cbb-4080-ac37-108ba4c3bfff None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:19.372446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:19.394025 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba8543d-e7ba-4fcf-879a-88909e608358 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:19.448848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:19.449503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.449754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.450034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.450736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] RBAC: Adding query filter params (type=network) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:19.451048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:19.452807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:19.460716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 415/839] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:19 2022] GET /identity/v3/services?type=network => generated 350 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:19.467067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:19.508546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:19.532470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5c868f-6cbb-4080-ac37-108ba4c3bfff None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:19.588972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:19.591171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.591655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.592151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.602335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:19.604260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:19.620317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:19.621296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:19.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 425/840] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 13:57:19 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:19.627568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:19.666890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f2c51b-b4b2-44f5-afcb-4d5c005a3ca0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:19.710195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64489ad0-8381-4dce-9964-4646123f3d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:19.711072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64489ad0-8381-4dce-9964-4646123f3d43 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.711403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64489ad0-8381-4dce-9964-4646123f3d43 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.711680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64489ad0-8381-4dce-9964-4646123f3d43 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.713810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64489ad0-8381-4dce-9964-4646123f3d43 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:19.715187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64489ad0-8381-4dce-9964-4646123f3d43 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:19.742462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 416/841] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 13:57:19 2022] POST /identity/v3/endpoints => generated 337 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:57:19.743474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64489ad0-8381-4dce-9964-4646123f3d43 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:19.754462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:19.780527 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e095eb-06e1-4d53-8c2c-793f0ed8597f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:19.827095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:19.828258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.828760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.829313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.830816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:19.831326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:19.833089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:19.848503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:19.858483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 426/842] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 15 13:57:19 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:19.873911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:19.882038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64489ad0-8381-4dce-9964-4646123f3d43 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:19.930978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d4e0c8-a7ad-47d6-8429-facea59b20f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:19.931659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4e0c8-a7ad-47d6-8429-facea59b20f2 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:19.931984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4e0c8-a7ad-47d6-8429-facea59b20f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:19.932258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4e0c8-a7ad-47d6-8429-facea59b20f2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:19.963339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d4e0c8-a7ad-47d6-8429-facea59b20f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-d7d4e0c8-a7ad-47d6-8429-facea59b20f2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:19.978796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:19.979860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:19.980712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 417/843] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:19.987372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d4e0c8-a7ad-47d6-8429-facea59b20f2 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.023994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f7cfb2-21d0-44f1-9fd7-28aa61c16081 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.069705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba73350-f25d-4029-8c58-9b28bf482b9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.069980 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba73350-f25d-4029-8c58-9b28bf482b9e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.070254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba73350-f25d-4029-8c58-9b28bf482b9e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.070520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba73350-f25d-4029-8c58-9b28bf482b9e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.074109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:20.151276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d4e0c8-a7ad-47d6-8429-facea59b20f2 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.165929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba73350-f25d-4029-8c58-9b28bf482b9e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.167483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba73350-f25d-4029-8c58-9b28bf482b9e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.192814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b490f5a-8b29-4558-b3a7-d9b922c985e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.193429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b490f5a-8b29-4558-b3a7-d9b922c985e7 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.193979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b490f5a-8b29-4558-b3a7-d9b922c985e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.193979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b490f5a-8b29-4558-b3a7-d9b922c985e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.194627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b490f5a-8b29-4558-b3a7-d9b922c985e7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:20.194933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b490f5a-8b29-4558-b3a7-d9b922c985e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.196319 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b490f5a-8b29-4558-b3a7-d9b922c985e7 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.198686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:20.207740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 418/844] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:20.209012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24957de-d1da-4bba-ac53-1265032c40b4 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.209313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24957de-d1da-4bba-ac53-1265032c40b4 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.209584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24957de-d1da-4bba-ac53-1265032c40b4 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.210897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 419/845] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:20.213324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24957de-d1da-4bba-ac53-1265032c40b4 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.282402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 427/846] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:19 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 15 13:57:20.284077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8980a71-cf58-4f0a-81d9-2149d318d6d2 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.284539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8980a71-cf58-4f0a-81d9-2149d318d6d2 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.284957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8980a71-cf58-4f0a-81d9-2149d318d6d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.307372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c8980a71-cf58-4f0a-81d9-2149d318d6d2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:20.308612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c8980a71-cf58-4f0a-81d9-2149d318d6d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:20.329180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24957de-d1da-4bba-ac53-1265032c40b4 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.337468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:20.366096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.366730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.367011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.367283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.379137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.380624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.391707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 428/847] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:20 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:20.391707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8980a71-cf58-4f0a-81d9-2149d318d6d2 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:20.406226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:20.407245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:20.408817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:20.408817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:20.408817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:20.408817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 420/848] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:20.413388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.476378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:20.502629 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8980a71-cf58-4f0a-81d9-2149d318d6d2 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.540782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb42295-83dc-4dd1-83fe-f87b83c88ad6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.554449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.555571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.556028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.556986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.570730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.573050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.579096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.579716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.580647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.581022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.581726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:20.582089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.583575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:20.596723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:20.598808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:20.605983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:20.605983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:20.605983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:20.605983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 429/849] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:20.606491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.611845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 421/850] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:20.616822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.628476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:20.712836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3722800c-bea2-4829-9603-92797e56e039 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.726774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da53703-7c63-46dd-9698-7baa9c7c7d01 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.737864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:20.750104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.750712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.750991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.751261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.752154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:20.752607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.753979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.762268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.763603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.763910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.764183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.766293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 430/851] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:20.766886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.776986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.778411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:20.790492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:20.791483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 422/852] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/users/swift => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:20.792490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.852495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:20.876297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011e4d06-41ec-4a47-b552-148aa4f25eaf None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.897082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4a4c7f-47cb-4168-ac8f-a9e2a99a265c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:20.912545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.913203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.913510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.913769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.914392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.915808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:20.924906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:20.925976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:20.927090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 431/853] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:20.928486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.938638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:20.939812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:20.940097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:20.940429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:20.947426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:20.948907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:20.960324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:20.961215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 423/854] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:20.962753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:20.995869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:21.034333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7641a-f16b-46be-b381-7d900dd9a0c5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.069394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:21.069926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.070205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.070303 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed40cc05-2afc-4bf8-abb6-5ca936909da4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.070472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.071201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:21.071530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:21.072975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:21.082425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:21.091511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 432/855] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:21.092416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:21.109877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:21.110569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.110921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.111833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.112496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:21.112863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:21.114285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:21.119737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 424/856] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:20 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:21.121199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:21.199564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371165b8-cecf-4dbc-8f34-d1ea648ae48e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.221738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:21.230389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b89b9-c2dc-4cb7-95ea-7b3ca809fa61 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.233946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:21.234544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.234818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.235089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.235806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:21.236633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:21.238203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:21.250508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 433/857] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:21.251466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:21.268894 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768d02bc-68d3-4481-a862-73ce5781b3a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:21.269507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768d02bc-68d3-4481-a862-73ce5781b3a6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.269789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768d02bc-68d3-4481-a862-73ce5781b3a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.270066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768d02bc-68d3-4481-a862-73ce5781b3a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.329380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:21.350723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768d02bc-68d3-4481-a862-73ce5781b3a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:21.352626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768d02bc-68d3-4481-a862-73ce5781b3a6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:21.379375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d98c2-9793-4345-badf-62c9037d042d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.415224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:21.415875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.416160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.416433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.417195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:21.417485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:21.418979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:21.431714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 434/858] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:21 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:21.431714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:21.454263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:21.464420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 425/859] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:21 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:57:21.465918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0ca0d5-d048-4bbf-8e3d-7283797b5e48 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.466353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0ca0d5-d048-4bbf-8e3d-7283797b5e48 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.466717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0ca0d5-d048-4bbf-8e3d-7283797b5e48 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.468423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 426/860] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:21.470047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0ca0d5-d048-4bbf-8e3d-7283797b5e48 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:21.554165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f798a-19de-4fbb-a847-2033fa519027 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.582033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:21.587436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0ca0d5-d048-4bbf-8e3d-7283797b5e48 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.594853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f27060-f522-45e7-aa93-13175757ae10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:21.596300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f27060-f522-45e7-aa93-13175757ae10 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.596808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f27060-f522-45e7-aa93-13175757ae10 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.597084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f27060-f522-45e7-aa93-13175757ae10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.597825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f27060-f522-45e7-aa93-13175757ae10 None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=75c8381b6ad84cac881db621d2d99af7) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:21.598138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f27060-f522-45e7-aa93-13175757ae10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:21.599727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f27060-f522-45e7-aa93-13175757ae10 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:21.606875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 435/861] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:21 2022] GET /identity/v3/role_assignments?user.id=75c8381b6ad84cac881db621d2d99af7&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:21.607567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c941b-3ee9-4ec4-90e6-f60fa5addea0 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.607938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c941b-3ee9-4ec4-90e6-f60fa5addea0 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.608218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c941b-3ee9-4ec4-90e6-f60fa5addea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.628748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d67c941b-3ee9-4ec4-90e6-f60fa5addea0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:21.629826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d67c941b-3ee9-4ec4-90e6-f60fa5addea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:21.641460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:21.641460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.642052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.642392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.657440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:21.659647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:21.679714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:21.680710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:21.681589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:21.681589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:21.681589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:21.681589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 427/862] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:21.683651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:21.725950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 436/863] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:21 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:21.746638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:21.807555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2cbc84-d43a-4294-8434-f63d31d1fc5a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.843594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:21.844164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:21.844449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:21.844727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:21.845400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:21.845841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:21.847244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:21.862229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 428/864] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:21.867560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67c941b-3ee9-4ec4-90e6-f60fa5addea0 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:21.976386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67c941b-3ee9-4ec4-90e6-f60fa5addea0 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:21.994156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:22.003078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:22.012847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:22.013877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:22.013877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:22.014132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:22.022331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:22.023822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:22.038870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:22.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:22.040706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 437/865] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:22.045775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:22.127748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc8c4b1-b947-4116-9e37-99c52614a031 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:22.138681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:22.174709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:22.175341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:22.175621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:22.175921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:22.176764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:22.176905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582db8c2-71df-4a48-a231-0383e5c49ccc None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:22.178129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:22.190247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:22.191441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:22.192557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 429/866] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Sep 15 13:57:22 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:22.200877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:22.223095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:22.223897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:22.224265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:22.224636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:22.225455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:22.225853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:22.227502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:22.246002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 438/867] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:22.248414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:22.287214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:22.338877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3884f-c1ca-4815-b441-d2099c8787cc None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:22.359051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d159b5ee-9e94-4ff3-b5a3-a87ee043e6ad None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:22.383950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:22.384494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:22.384773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:22.385044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:22.385878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:22.386182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:22.388335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:22.395927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 430/868] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:22 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:22.401805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:22.402829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b38884-98ca-4bc3-8697-6d01e6ee5b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:22.403453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b38884-98ca-4bc3-8697-6d01e6ee5b10 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:22.403750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b38884-98ca-4bc3-8697-6d01e6ee5b10 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:22.404024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b38884-98ca-4bc3-8697-6d01e6ee5b10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:22.404751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b38884-98ca-4bc3-8697-6d01e6ee5b10 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=30fc2a26b4fc4d24b984fa8097752055) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:22.405084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b38884-98ca-4bc3-8697-6d01e6ee5b10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:22.407864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b38884-98ca-4bc3-8697-6d01e6ee5b10 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:22.419786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 439/869] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:22 2022] GET /identity/v3/role_assignments?user.id=30fc2a26b4fc4d24b984fa8097752055&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:22.449675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:22.530247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034c64d4-c2fe-4e0c-a9a1-c47371789f22 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:22.569915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7d337f-9be9-4ce3-989b-d02fad8971b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:22.570791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d337f-9be9-4ce3-989b-d02fad8971b9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:22.571243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d337f-9be9-4ce3-989b-d02fad8971b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:22.571656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d337f-9be9-4ce3-989b-d02fad8971b9 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:22.572464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7d337f-9be9-4ce3-989b-d02fad8971b9 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:22.572908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7d337f-9be9-4ce3-989b-d02fad8971b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:22.574420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7d337f-9be9-4ce3-989b-d02fad8971b9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:22.581120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 431/870] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:22 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:23.708409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2039ced-e33a-4270-b009-303aa97e34e6 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:23.709340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2039ced-e33a-4270-b009-303aa97e34e6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:23.709775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2039ced-e33a-4270-b009-303aa97e34e6 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:23.711515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 440/871] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:23.717187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f608b218-608f-4809-ba23-651304fffad1 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:23.718061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f608b218-608f-4809-ba23-651304fffad1 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:23.718545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f608b218-608f-4809-ba23-651304fffad1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:23.737966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f608b218-608f-4809-ba23-651304fffad1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:23.739505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f608b218-608f-4809-ba23-651304fffad1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:23.810587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 432/872] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:23 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:23.996955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd1693-7a24-45f6-b579-de13e91dae99 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:23.997144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd1693-7a24-45f6-b579-de13e91dae99 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:23.997474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd1693-7a24-45f6-b579-de13e91dae99 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:23.998960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 441/873] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:24.004727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184de46-9549-4dd7-976b-429aef2f6299 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.005249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184de46-9549-4dd7-976b-429aef2f6299 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.005737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184de46-9549-4dd7-976b-429aef2f6299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.025045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4184de46-9549-4dd7-976b-429aef2f6299 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:24.026514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4184de46-9549-4dd7-976b-429aef2f6299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:24.050402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:24.088108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd1693-7a24-45f6-b579-de13e91dae99 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:24.100871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 433/874] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:24 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:24.195765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:24.200105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd1693-7a24-45f6-b579-de13e91dae99 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:24.235901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eada74-7cb4-4cd1-883f-d7a2e155eacb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:24.237799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eada74-7cb4-4cd1-883f-d7a2e155eacb None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.238278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eada74-7cb4-4cd1-883f-d7a2e155eacb None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.238701 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eada74-7cb4-4cd1-883f-d7a2e155eacb None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.239570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eada74-7cb4-4cd1-883f-d7a2e155eacb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:24.241156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eada74-7cb4-4cd1-883f-d7a2e155eacb None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:24.259798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 442/875] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:57:24 2022] POST /identity/v3/services => generated 251 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:57:24.374669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4184de46-9549-4dd7-976b-429aef2f6299 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:24.490977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4184de46-9549-4dd7-976b-429aef2f6299 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:24.534341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf017296-2f91-490d-ae8e-a3d3ddd7c034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:24.534870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf017296-2f91-490d-ae8e-a3d3ddd7c034 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.535148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf017296-2f91-490d-ae8e-a3d3ddd7c034 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.535414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf017296-2f91-490d-ae8e-a3d3ddd7c034 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.545846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf017296-2f91-490d-ae8e-a3d3ddd7c034 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-cf017296-2f91-490d-ae8e-a3d3ddd7c034 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:24.550784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:24.552260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:24.553166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 434/876] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 15 13:57:24 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:24.558443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eada74-7cb4-4cd1-883f-d7a2e155eacb None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:24.668520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eada74-7cb4-4cd1-883f-d7a2e155eacb None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:24.710462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e8c4de-d3a0-4b3b-ac67-9674ca0e2f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:24.712009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e8c4de-d3a0-4b3b-ac67-9674ca0e2f05 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.712499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e8c4de-d3a0-4b3b-ac67-9674ca0e2f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.712971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e8c4de-d3a0-4b3b-ac67-9674ca0e2f05 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.713844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e8c4de-d3a0-4b3b-ac67-9674ca0e2f05 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:24.714372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e8c4de-d3a0-4b3b-ac67-9674ca0e2f05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:24.715982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e8c4de-d3a0-4b3b-ac67-9674ca0e2f05 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:24.725858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 443/877] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 15 13:57:24 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:24.731739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf017296-2f91-490d-ae8e-a3d3ddd7c034 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:24.749453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:24.756449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea0e8b-478a-4ce4-a4c1-ea62ed4b3fcf None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.756737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea0e8b-478a-4ce4-a4c1-ea62ed4b3fcf None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.757019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea0e8b-478a-4ce4-a4c1-ea62ed4b3fcf None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.758595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 444/878] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:24.763557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a410bd-a132-4c93-bd5a-57a09b0bf8bb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.763870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a410bd-a132-4c93-bd5a-57a09b0bf8bb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.764148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a410bd-a132-4c93-bd5a-57a09b0bf8bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.783203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-b0a410bd-a132-4c93-bd5a-57a09b0bf8bb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:24.784486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b0a410bd-a132-4c93-bd5a-57a09b0bf8bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:24.807073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:24.844770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf017296-2f91-490d-ae8e-a3d3ddd7c034 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:24.865675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 445/879] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:24 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:24.868988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a057956-9b3a-41a4-bfc3-17a3254aa87e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.869271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a057956-9b3a-41a4-bfc3-17a3254aa87e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.869543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a057956-9b3a-41a4-bfc3-17a3254aa87e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.870933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 446/880] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:24.876041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39caa4f5-971e-4415-b782-cd70a81cd984 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.876321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39caa4f5-971e-4415-b782-cd70a81cd984 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.876665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39caa4f5-971e-4415-b782-cd70a81cd984 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.888846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:24.889903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:24.890396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:24.890888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:24.896325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-39caa4f5-971e-4415-b782-cd70a81cd984 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:24.897497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-39caa4f5-971e-4415-b782-cd70a81cd984 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:24.903665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:24.905898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:24.925491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:24.926786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:24.927733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:24.927733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:24.927733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:24.933830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 435/881] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:24.935328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:24.995756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 447/882] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:24 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:25.012780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:25.049966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bfda13-b505-4a80-9949-af2f269198f4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:25.085477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:25.086450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:25.086949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:25.087447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:25.088327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:25.088853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:25.090431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:25.107562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 436/883] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:25.109289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39caa4f5-971e-4415-b782-cd70a81cd984 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:25.141553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:25.201736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:25.221828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39caa4f5-971e-4415-b782-cd70a81cd984 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:25.259526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:25.260242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:25.260521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:25.260794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:25.268557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858063f5-7d4b-42e4-859c-0c4ddf2c9468 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:25.269587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:25.271221 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:25.285657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:25.286673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:25.287647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 448/884] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 13:57:25 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:25.288306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:25.304634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:25.319936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:25.320578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:25.320860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:25.321140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:25.321731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:25.323271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:25.332407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:25.333657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:25.334495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 437/885] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:25 2022] GET /identity/v3/services/object-store => generated 93 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:25.335301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:25.401226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3902c8-25b9-4da5-a2d3-cf9078e6f301 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:25.434505 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c2f63a-3ff9-4d88-8e0d-b90c370c9d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:25.435351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c2f63a-3ff9-4d88-8e0d-b90c370c9d42 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:25.435823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c2f63a-3ff9-4d88-8e0d-b90c370c9d42 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:25.436251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c2f63a-3ff9-4d88-8e0d-b90c370c9d42 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:25.437050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c2f63a-3ff9-4d88-8e0d-b90c370c9d42 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:25.438557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c2f63a-3ff9-4d88-8e0d-b90c370c9d42 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:25.442140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f0d9c1-a057-45ee-b219-11dc6896c407 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:25.455454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 449/886] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:57:25 2022] POST /identity/v3/services => generated 269 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:25.456238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c2f63a-3ff9-4d88-8e0d-b90c370c9d42 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:25.458916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:25.509824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:25.511518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:25.512085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:25.512544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:25.514043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:25.514632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:25.516981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:25.544494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 438/887] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 15 13:57:25 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:25.553291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:25.586076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c2f63a-3ff9-4d88-8e0d-b90c370c9d42 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:25.610654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:25.629531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:25.630124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:25.630408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:25.630683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:25.631330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:25.631638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:25.633051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:25.639009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 450/888] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:25 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:25.644604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:25.688464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e8b34-0314-42f3-8bba-48586acb2d55 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:25.730550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:25.737985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc0f3ae-03bc-4fb3-a9ed-16c52046e9af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:25.738794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc0f3ae-03bc-4fb3-a9ed-16c52046e9af None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:25.739082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc0f3ae-03bc-4fb3-a9ed-16c52046e9af None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:25.739362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc0f3ae-03bc-4fb3-a9ed-16c52046e9af None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:25.774535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e51ad7-140e-43eb-9575-10f179cd88ce None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:25.819080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:25.819710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:25.820016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:25.820291 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:25.820988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:25.821290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:25.822728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:25.827853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 451/889] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:25 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:25.837294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc0f3ae-03bc-4fb3-a9ed-16c52046e9af None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:25.838699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc0f3ae-03bc-4fb3-a9ed-16c52046e9af None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:25.888038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:25.950184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 439/890] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:25 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:57:25.956898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:26.072172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8453187b-1d6d-499e-b04e-ba88e2b2e703 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:26.102075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:26.108238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:26.108881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:26.109163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:26.109473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:26.121400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:26.122866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:26.138070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:26.139017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:26.140427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:26.140427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:26.140427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:26.140427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 452/891] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:26.145642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc0f3ae-03bc-4fb3-a9ed-16c52046e9af None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:26.249503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc0f3ae-03bc-4fb3-a9ed-16c52046e9af None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:26.289774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:26.290389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:26.290671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:26.290948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:26.291596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:26.291931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:26.293287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:26.306400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 440/892] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:26.312018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:26.422855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5068793-0d2c-4997-819f-bae4ffa1eba4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:26.461433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92401a38-0ba4-4752-a25f-94665e4dd146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:26.462075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92401a38-0ba4-4752-a25f-94665e4dd146 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:26.462357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92401a38-0ba4-4752-a25f-94665e4dd146 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:26.462630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92401a38-0ba4-4752-a25f-94665e4dd146 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:26.469759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92401a38-0ba4-4752-a25f-94665e4dd146 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:26.471455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92401a38-0ba4-4752-a25f-94665e4dd146 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-92401a38-0ba4-4752-a25f-94665e4dd146 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:26.482858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:26.483779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:26.484664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 453/893] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:26.492244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:26.504099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:26.602990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a3fb33-abf3-4c2d-8077-bd138f38b641 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:26.622160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbad1bb-b7d3-4725-82c4-445f78e9f370 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:26.622494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbad1bb-b7d3-4725-82c4-445f78e9f370 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:26.622723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbad1bb-b7d3-4725-82c4-445f78e9f370 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:26.624326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 454/894] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:26.629506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e10c4b5-50fc-4541-acf7-4f527d2eaaa6 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:26.629787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e10c4b5-50fc-4541-acf7-4f527d2eaaa6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:26.630057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e10c4b5-50fc-4541-acf7-4f527d2eaaa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:26.641976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:26.642370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:26.642781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:26.643267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:26.644047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:26.644353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:26.645786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:26.649854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3e10c4b5-50fc-4541-acf7-4f527d2eaaa6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:26.651048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3e10c4b5-50fc-4541-acf7-4f527d2eaaa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:26.660290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 441/895] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:26.668701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:26.695798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:26.743599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 455/896] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:26 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:26.750201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:26.793279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed4570c-686a-47bf-8c49-0300c7651a8f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:26.829239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:26.830818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:26.831300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:26.831810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:26.832743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] RBAC: Adding query filter params (role.id=d511c5452170401cba1f1162075b62be, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=75c8381b6ad84cac881db621d2d99af7) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:26.833251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:26.834873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:26.841505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 442/897] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:26 2022] GET /identity/v3/role_assignments?user.id=75c8381b6ad84cac881db621d2d99af7&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=d511c5452170401cba1f1162075b62be => generated 259 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:27.031524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e10c4b5-50fc-4541-acf7-4f527d2eaaa6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:27.134783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e10c4b5-50fc-4541-acf7-4f527d2eaaa6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:27.170391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:27.171024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:27.171298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:27.171563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:27.172171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:27.174130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:27.183480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:27.184540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:27.185402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 456/898] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:27 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:27.191017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:27.295066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e39b65-a10f-4322-82cb-efee0ef799d3 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:27.332076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:27.338593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:27.339202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:27.339479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:27.339772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:27.340404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:27.340705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:27.342055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:27.347876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 443/899] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:27 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:27.353176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:27.391064 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:27.460587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a4608d-ac54-494e-a491-c66f647141c7 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:27.495392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:27.496037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:27.496313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:27.496586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:27.497220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:27.497516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:27.498884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:27.504959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 457/900] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:27 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:27.510708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:27.616713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b5609e-253b-4770-be4c-c03dcff85b38 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:27.659286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9e0dc-0354-416f-9715-e833da9bc0a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:27.660014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9e0dc-0354-416f-9715-e833da9bc0a0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:27.660294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9e0dc-0354-416f-9715-e833da9bc0a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:27.660566 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9e0dc-0354-416f-9715-e833da9bc0a0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:27.661293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d9e0dc-0354-416f-9715-e833da9bc0a0 None admin] RBAC: Adding query filter params (interface=public, service_id=fa856318b190494e8b4d52074b781927, region_id=RegionOne) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:27.661602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d9e0dc-0354-416f-9715-e833da9bc0a0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:27.663006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d9e0dc-0354-416f-9715-e833da9bc0a0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:27.669232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 444/901] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 15 13:57:27 2022] GET /identity/v3/endpoints?service_id=fa856318b190494e8b4d52074b781927&interface=public®ion_id=RegionOne => generated 194 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:27.675181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:27.761056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2c599-edb7-42ae-90c9-ac77cc269b84 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:27.761703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2c599-edb7-42ae-90c9-ac77cc269b84 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:27.761703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2c599-edb7-42ae-90c9-ac77cc269b84 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:27.763166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 445/902] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:27.769326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494f9268-54ba-4971-b11e-b87e72c973e0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:27.769612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494f9268-54ba-4971-b11e-b87e72c973e0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:27.770039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494f9268-54ba-4971-b11e-b87e72c973e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:27.785581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:27.788626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7764ae-2ef1-4d77-b742-305f6bf25ed4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:27.793058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-494f9268-54ba-4971-b11e-b87e72c973e0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:27.794489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-494f9268-54ba-4971-b11e-b87e72c973e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:27.828527 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e88c76-5f6a-4526-b597-d618d2ed4e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:27.829149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e88c76-5f6a-4526-b597-d618d2ed4e00 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:27.829432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e88c76-5f6a-4526-b597-d618d2ed4e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:27.830246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e88c76-5f6a-4526-b597-d618d2ed4e00 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:27.830844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e88c76-5f6a-4526-b597-d618d2ed4e00 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:27.832871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e88c76-5f6a-4526-b597-d618d2ed4e00 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:27.843715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 458/903] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 13:57:27 2022] GET /identity/v3/services => generated 1268 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 13:57:27.872147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:27.887871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 446/904] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:27 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:28.161211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e88c76-5f6a-4526-b597-d618d2ed4e00 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:28.213836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1c3e8-abd5-45fd-b797-e3d031af6abe None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:28.214141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1c3e8-abd5-45fd-b797-e3d031af6abe None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:28.214419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b1c3e8-abd5-45fd-b797-e3d031af6abe None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:28.216968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 447/905] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:28.226782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97f367e-7de1-4332-8e04-9096e86ac60f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:28.227331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97f367e-7de1-4332-8e04-9096e86ac60f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:28.227611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97f367e-7de1-4332-8e04-9096e86ac60f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:28.249067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d97f367e-7de1-4332-8e04-9096e86ac60f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:28.250937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d97f367e-7de1-4332-8e04-9096e86ac60f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:28.274587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e88c76-5f6a-4526-b597-d618d2ed4e00 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:28.323647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:28.324505 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:28.324959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:28.325397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:28.326203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:28.327577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:28.337160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:28.338117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:28.339017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:28.339800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 448/907] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:28 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:28.343001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 459/907] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Sep 15 13:57:28 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:28.344427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97f367e-7de1-4332-8e04-9096e86ac60f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:28.451131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97f367e-7de1-4332-8e04-9096e86ac60f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:28.473949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:28.492340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:28.492946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:28.493227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:28.493495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:28.494116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:28.494413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:28.495787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:28.500695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 449/908] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:28 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:28.509072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:28.540337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:28.620390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e1752b-2830-445b-aef7-93599a5d13c5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:28.658876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:28.660616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:28.661305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:28.661591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:28.661868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:28.662517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:28.662830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:28.664326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:28.675157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 460/909] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:28 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:28.676966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:28.776287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:28.778006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68c047-e970-4637-83bd-fd9e7bcc88bd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:28.785996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd4c091-11d7-4577-8cfe-c21f0d0498a8 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:28.825215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f3a8b9-eaf7-4839-bcf2-3b49248e80b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:28.826181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f3a8b9-eaf7-4839-bcf2-3b49248e80b6 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:28.826645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f3a8b9-eaf7-4839-bcf2-3b49248e80b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:28.827108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f3a8b9-eaf7-4839-bcf2-3b49248e80b6 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:28.828460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f3a8b9-eaf7-4839-bcf2-3b49248e80b6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:28.830564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f3a8b9-eaf7-4839-bcf2-3b49248e80b6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:28.833383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:28.833986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:28.834262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:28.834532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:28.835411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] RBAC: Adding query filter params (interface=public, service_id=956c724c805841169f6e49d6fc55ca4d, region_id=RegionOne) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:28.835740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:28.837835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:28.843329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 461/910] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 15 13:57:28 2022] GET /identity/v3/endpoints?service_id=956c724c805841169f6e49d6fc55ca4d&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:28.850888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:28.854033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 450/911] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:57:28 2022] POST /identity/v3/services => generated 241 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:57:28.979022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6d389c-90fe-4f75-bb59-1262f71feb07 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:29.028765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da477f1e-1f3d-4d6a-84f3-cf99d47d5fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:29.029363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da477f1e-1f3d-4d6a-84f3-cf99d47d5fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:29.029685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da477f1e-1f3d-4d6a-84f3-cf99d47d5fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:29.029960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da477f1e-1f3d-4d6a-84f3-cf99d47d5fb9 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:29.030568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da477f1e-1f3d-4d6a-84f3-cf99d47d5fb9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:29.031976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da477f1e-1f3d-4d6a-84f3-cf99d47d5fb9 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:29.040099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 462/912] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 13:57:28 2022] GET /identity/v3/services => generated 1497 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 13:57:29.055327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d42c3-4dd9-4cfa-a339-4d5463a0caf3 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:29.055642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d42c3-4dd9-4cfa-a339-4d5463a0caf3 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:29.055920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59d42c3-4dd9-4cfa-a339-4d5463a0caf3 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:29.057627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 451/913] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:29.063412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30521c4-c1e2-48bc-b8a1-8dad8d05fa50 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:29.063791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30521c4-c1e2-48bc-b8a1-8dad8d05fa50 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:29.063995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30521c4-c1e2-48bc-b8a1-8dad8d05fa50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:29.085918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-b30521c4-c1e2-48bc-b8a1-8dad8d05fa50 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:29.087499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b30521c4-c1e2-48bc-b8a1-8dad8d05fa50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:29.101245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:29.172128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 463/914] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:29 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:29.443781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59d42c3-4dd9-4cfa-a339-4d5463a0caf3 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:29.492874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:29.556366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59d42c3-4dd9-4cfa-a339-4d5463a0caf3 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:29.591773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a0c67b-e016-4974-8c65-0f5c837e5ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:29.592339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a0c67b-e016-4974-8c65-0f5c837e5ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:29.592612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a0c67b-e016-4974-8c65-0f5c837e5ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:29.592882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a0c67b-e016-4974-8c65-0f5c837e5ac4 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:29.604040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a0c67b-e016-4974-8c65-0f5c837e5ac4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-f2a0c67b-e016-4974-8c65-0f5c837e5ac4 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:29.608730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:29.609684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:29.611630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 452/915] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 15 13:57:29 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:29.616706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30521c4-c1e2-48bc-b8a1-8dad8d05fa50 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:29.719677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30521c4-c1e2-48bc-b8a1-8dad8d05fa50 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:29.755956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:29.756568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:29.756848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:29.757118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:29.757783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:29.758082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:29.759460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:29.768419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 464/916] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 15 13:57:29 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:29.774032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a0c67b-e016-4974-8c65-0f5c837e5ac4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:29.877905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:29.880823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a0c67b-e016-4974-8c65-0f5c837e5ac4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:29.917362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:29.917994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:29.918280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:29.918549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:29.930171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:29.931610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:29.947119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:29.948126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:29.949255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:29.949255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:29.949255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:29.949255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 453/917] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:29.954191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:29.977306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:30.066242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f63af4a-08ac-449c-a366-5cb14575d2d5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:30.101349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ced2ac-3a63-4be3-89c8-e401027079b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:30.101966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ced2ac-3a63-4be3-89c8-e401027079b2 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.102246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ced2ac-3a63-4be3-89c8-e401027079b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.102532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ced2ac-3a63-4be3-89c8-e401027079b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.103159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ced2ac-3a63-4be3-89c8-e401027079b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:30.103461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ced2ac-3a63-4be3-89c8-e401027079b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:30.104857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ced2ac-3a63-4be3-89c8-e401027079b2 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:30.118260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 465/918] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:30.123849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:30.126731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79adc848-e5fa-4283-b263-e45da5b57878 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.127186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79adc848-e5fa-4283-b263-e45da5b57878 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.127699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79adc848-e5fa-4283-b263-e45da5b57878 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.131771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 466/919] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:30.136002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8cf23d-be12-4927-9a02-135d423dc7e6 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.136281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8cf23d-be12-4927-9a02-135d423dc7e6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.136551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8cf23d-be12-4927-9a02-135d423dc7e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.155826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-aa8cf23d-be12-4927-9a02-135d423dc7e6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:30.157001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-aa8cf23d-be12-4927-9a02-135d423dc7e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:30.237095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:30.251719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f515e83-ac2b-4d4d-9ad3-24bd086beada None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:30.252384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 467/920] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:30 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:30.280676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:30.281295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.281582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.281839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.291349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:30.292852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:30.305676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:30.306621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:30.307552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 454/921] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 13:57:30 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:30.315732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8cf23d-be12-4927-9a02-135d423dc7e6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:30.422150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8cf23d-be12-4927-9a02-135d423dc7e6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:30.458422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:30.459351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.459860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.460356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.461288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:30.461810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:30.463429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:30.481889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 468/922] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 15 13:57:30 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:30.488177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:30.508830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:30.535214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:30.595143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:30.604378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4099c991-c605-4a9e-9b75-6c8d907fb0a3 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:30.640190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:30.641226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.641707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.642216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.643065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbde79be-a31b-4124-bd49-f0f548a70bcb None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:30.653933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:30.655665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:30.671631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:30.672681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:30.673675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:30.673675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:30.673675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:30.676104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 455/923] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:30.681622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:30.694154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:30.695292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.695817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.696292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.697169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:30.698809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:30.709062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:30.710109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 15 13:57:30.710948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:30.718589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 469/924] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:30 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:30.721477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:30.749245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:30.790480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58afae5-faad-40ef-bf5c-acfaedd411c9 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:30.825435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:30.826435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.826933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.827405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.828304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:30.828810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:30.831056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:30.837412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21165810-d771-49cf-a9e2-220fe5e1eb37 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:30.844182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 456/925] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:30.850933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:30.867429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:30.886147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:30.886792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:30.887101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:30.887386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:30.888334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:30.888645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:30.890246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:30.901788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 470/926] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:30 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:30.902867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:30.974675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4a9c57-9a9a-4148-9384-b84505e58d41 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:30.998807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:31.010112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:31.010969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.011082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.011275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.018783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:31.020266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:31.022195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446bd84b-7263-44dc-8757-cd8162a79c96 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:31.031884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:31.033413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:31.034299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:31.034299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:31.034299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:31.034299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:31.034299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:31.034299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:31.034299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:31.034299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:31.038402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 457/927] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:31.039333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:31.063819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:31.064978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.065594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.066069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.066907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:31.067419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:31.069152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:31.080251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 471/928] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 15 13:57:30 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:31.081764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:31.140635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:31.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfb07b7-480e-4bb0-a578-b4afeb0162ff None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:31.184216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:31.184802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.185082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.185355 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.186083 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:31.186610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:31.187900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:31.205696 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06bee63-b38f-4f46-9bb0-13bde49df862 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:31.208693 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 458/929] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:31.210683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:31.243596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fdb927-b7fd-4958-9180-c0ec1d3aea21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:31.244178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fdb927-b7fd-4958-9180-c0ec1d3aea21 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.244434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fdb927-b7fd-4958-9180-c0ec1d3aea21 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.244704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fdb927-b7fd-4958-9180-c0ec1d3aea21 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.245388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fdb927-b7fd-4958-9180-c0ec1d3aea21 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:31.246746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fdb927-b7fd-4958-9180-c0ec1d3aea21 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:31.260104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:31.269359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 472/930] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 13:57:31 2022] POST /identity/v3/endpoints => generated 346 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:31.270910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0a86b-2ae1-4663-add7-51da1fb13349 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.271435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0a86b-2ae1-4663-add7-51da1fb13349 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.271943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0a86b-2ae1-4663-add7-51da1fb13349 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.277534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 473/931] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:31.279119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c83d14-1fa3-42cd-bb99-d09349bacafb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.279405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c83d14-1fa3-42cd-bb99-d09349bacafb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.279679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c83d14-1fa3-42cd-bb99-d09349bacafb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.304347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c7c83d14-1fa3-42cd-bb99-d09349bacafb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:31.306641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c7c83d14-1fa3-42cd-bb99-d09349bacafb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:31.344822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ad314f-5902-475a-810d-e619f378d359 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:31.391080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39264acf-8af5-4a4e-89ad-d541e773ef73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:31.391781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39264acf-8af5-4a4e-89ad-d541e773ef73 None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.392067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39264acf-8af5-4a4e-89ad-d541e773ef73 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.392347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39264acf-8af5-4a4e-89ad-d541e773ef73 None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/75c8381b6ad84cac881db621d2d99af7/roles/d511c5452170401cba1f1162075b62be` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.394589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39264acf-8af5-4a4e-89ad-d541e773ef73 None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=75c8381b6ad84cac881db621d2d99af7, role_id=d511c5452170401cba1f1162075b62be)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:31.402733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39264acf-8af5-4a4e-89ad-d541e773ef73 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:31.402945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 474/932] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:31.402945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561be73-e621-42c1-9482-77c5cda53113 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.402945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561be73-e621-42c1-9482-77c5cda53113 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.402945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561be73-e621-42c1-9482-77c5cda53113 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.402945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 475/933] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:31.409830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf5ebd9-4e82-4530-8ce8-71d973f36e61 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.410163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf5ebd9-4e82-4530-8ce8-71d973f36e61 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.410903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf5ebd9-4e82-4530-8ce8-71d973f36e61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.416067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 459/934] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:57:31 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/75c8381b6ad84cac881db621d2d99af7/roles/d511c5452170401cba1f1162075b62be => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 13:57:31.441168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-fbf5ebd9-4e82-4530-8ce8-71d973f36e61 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:31.442699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-fbf5ebd9-4e82-4530-8ce8-71d973f36e61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:31.472853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:31.525996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 476/935] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:31.679364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39264acf-8af5-4a4e-89ad-d541e773ef73 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:31.780612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:31.784139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39264acf-8af5-4a4e-89ad-d541e773ef73 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:31.810080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf5ebd9-4e82-4530-8ce8-71d973f36e61 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:31.823802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:31.824560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.824845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.825234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.825910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:31.827264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:31.837360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:31.838393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:31.839238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 460/936] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:31 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:31.844612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:31.915546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf5ebd9-4e82-4530-8ce8-71d973f36e61 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:31.969150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5c6758-10ff-4ac6-968c-82d04797b58f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:31.972421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:31.973909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:31.974194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:31.974469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:31.975065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:31.976497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:31.988839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:31.989890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 15 13:57:31.990796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:31.995911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 477/937] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Sep 15 13:57:31 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:31.996834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:32.019444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:32.020096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:32.020379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:32.020900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:32.021546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:32.022252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:32.023622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:32.029203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:32.035800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 461/938] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:31 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:32.039243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:32.072058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:32.129207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efe4c49-7faf-4e5c-875c-db1678ae8a5e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:32.155054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ea4b58-3278-4283-aba8-d595a75cc5de None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:32.166846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:32.167830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:32.168315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:32.168801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:32.169744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:32.170370 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:32.171956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:32.178092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 478/939] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:31 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:32.184824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:32.198880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:32.199930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:32.200780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:32.201257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:32.202692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:32.203324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:32.205897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:32.216342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 462/940] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:32.219031 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:32.308427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:32.322361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4b71df-b544-4919-b98e-1d11bc4bc7f6 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:32.350275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:32.361921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d33d2b9-bae0-4bbf-9866-c5f6b8795c3c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:32.362347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:32.363252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:32.363918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:32.364401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:32.365368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:32.365890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:32.367509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:32.374097 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 479/941] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:32 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:32.380708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:32.411915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:32.412536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:32.412816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:32.413091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:32.414119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] RBAC: Adding query filter params (interface=public, service_id=37c34b7b61b240df8c9ea53d8475cb2e, region_id=RegionOne) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:32.414483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:32.416028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:32.423884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 463/942] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 15 13:57:32 2022] GET /identity/v3/endpoints?service_id=37c34b7b61b240df8c9ea53d8475cb2e&interface=public®ion_id=RegionOne => generated 194 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:32.430387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:32.504743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f48222e-6d93-4efc-b3a4-f45ef38c8523 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:32.537656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582f69e5-f458-4be2-8ce2-7817caa325bd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:32.540891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8a0855-01bc-4659-ad6f-f18c173a38e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:32.541722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a0855-01bc-4659-ad6f-f18c173a38e1 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:32.542224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a0855-01bc-4659-ad6f-f18c173a38e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:32.542692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8a0855-01bc-4659-ad6f-f18c173a38e1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:32.543572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8a0855-01bc-4659-ad6f-f18c173a38e1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:32.545158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8a0855-01bc-4659-ad6f-f18c173a38e1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:32.572568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 480/943] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 13:57:32 2022] POST /identity/v3/endpoints => generated 347 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:32.585237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d540bb-0e93-4a2d-96ed-cbb2b6ce5f4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:32.586324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d540bb-0e93-4a2d-96ed-cbb2b6ce5f4e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:32.586857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d540bb-0e93-4a2d-96ed-cbb2b6ce5f4e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:32.587350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d540bb-0e93-4a2d-96ed-cbb2b6ce5f4e None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:32.588234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d540bb-0e93-4a2d-96ed-cbb2b6ce5f4e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:32.589934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d540bb-0e93-4a2d-96ed-cbb2b6ce5f4e None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:32.604124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 464/944] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 13:57:32 2022] GET /identity/v3/services => generated 1497 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 13:57:33.576970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8940dd-0309-49c4-92e2-0c962b62e1c1 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:33.577463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8940dd-0309-49c4-92e2-0c962b62e1c1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:33.577463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8940dd-0309-49c4-92e2-0c962b62e1c1 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:33.578981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 481/945] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:33.585367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5386dcb-59c7-4b5a-af9a-b72f223ace72 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:33.585759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5386dcb-59c7-4b5a-af9a-b72f223ace72 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:33.585943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5386dcb-59c7-4b5a-af9a-b72f223ace72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:33.604042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d5386dcb-59c7-4b5a-af9a-b72f223ace72 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:33.605203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d5386dcb-59c7-4b5a-af9a-b72f223ace72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:33.616139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:33.667376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e39151-ba22-49f9-9dbd-9a2df4a27d73 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:33.667658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e39151-ba22-49f9-9dbd-9a2df4a27d73 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:33.667959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e39151-ba22-49f9-9dbd-9a2df4a27d73 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:33.669404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 482/946] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:33.674950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deabb99-ce12-4c52-96fa-1ef668866880 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:33.675233 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deabb99-ce12-4c52-96fa-1ef668866880 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:33.675577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deabb99-ce12-4c52-96fa-1ef668866880 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:33.685776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 465/947] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:33 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:33.696563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7deabb99-ce12-4c52-96fa-1ef668866880 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:33.697756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7deabb99-ce12-4c52-96fa-1ef668866880 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:33.699769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:33.768085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 483/948] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:33 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:33.967024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5386dcb-59c7-4b5a-af9a-b72f223ace72 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:34.040662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deabb99-ce12-4c52-96fa-1ef668866880 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:34.076635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5386dcb-59c7-4b5a-af9a-b72f223ace72 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:34.121503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:34.122561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.122854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.123175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.124268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:34.126241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:34.130146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING keystone.server.flask.common [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 15 13:57:34.148904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deabb99-ce12-4c52-96fa-1ef668866880 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:34.153195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.sql.core [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 15 13:57:34.153195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 15 13:57:34.153195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [parameters: {'id': '1ba8779e90b849809723e84cd99154e9', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 15 13:57:34.153195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81654) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1ba8779e90b849809723e84cd99154e9.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1ba8779e90b849809723e84cd99154e9. Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result = self._query(query) Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application conn.query(q) Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 15 13:57:34.174137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result.read() Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 15 13:57:34.175095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application next(self.gen) Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application yield resource Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application next(self.gen) Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.session.rollback() Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.force_reraise() Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise self.value Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application session.commit() Sep 15 13:57:34.176073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._prepare_impl() Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.session.flush() Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._flush(objects) Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application compat.raise_( Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application flush_context.execute() Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 15 13:57:34.177213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rec.execute(self) Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 15 13:57:34.178123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result = self._query(query) Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application conn.query(q) Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result.read() Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 15 13:57:34.178943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [parameters: {'id': '1ba8779e90b849809723e84cd99154e9', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:34.180953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:34.182253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 1ba8779e90b849809723e84cd99154e9. Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:34.183400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 466/949] 10.222.0.27 () {60 vars in 1140 bytes} [Thu Sep 15 13:57:33 2022] POST /identity/v3/projects => generated 262 bytes in 210 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:34.192174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:34.192174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:34.194659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9547d10-ed91-4e4b-8c14-af59d274700a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:34.195392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9547d10-ed91-4e4b-8c14-af59d274700a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.195894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9547d10-ed91-4e4b-8c14-af59d274700a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.196243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9547d10-ed91-4e4b-8c14-af59d274700a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.205091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:34.209569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9547d10-ed91-4e4b-8c14-af59d274700a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-b9547d10-ed91-4e4b-8c14-af59d274700a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:34.215938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:34.217832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:34.219193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 484/950] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 15 13:57:34 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:34.229555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9547d10-ed91-4e4b-8c14-af59d274700a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:34.306607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60cd199-489d-4a49-baf5-c3f188b34af0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:34.343727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:34.344375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.344651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.344921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.349197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9547d10-ed91-4e4b-8c14-af59d274700a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:34.352659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:34.354302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:34.366643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:34.368323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:34.369459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 467/951] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:34.373396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:34.396330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:34.397205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.397482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.397752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.398578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:34.398878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:34.400590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:34.415975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 485/952] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 15 13:57:34 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:34.418011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:34.461411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:34.487871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:34.510601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e66893-d261-423e-84eb-aeb781fa6d5e None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:34.543452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2e874-4140-4010-95f2-482d59d0038f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:34.549361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:34.550183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.550457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.550724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.551389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:34.551959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:34.553492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:34.575813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 468/953] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:34 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:34.590256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:34.590947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.595730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.595730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.604795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:34.606511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:34.623708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:34.625993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:34.628320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:34.628320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:34.628320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:34.628320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 486/954] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:34.631461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:34.755334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcd1b3e-88a9-4dbb-805e-93b68457383b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:34.793827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:34.798150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446942c3-e69a-4747-9314-ffef63197cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:34.799718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446942c3-e69a-4747-9314-ffef63197cb8 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.800456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446942c3-e69a-4747-9314-ffef63197cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.800743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446942c3-e69a-4747-9314-ffef63197cb8 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.801444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446942c3-e69a-4747-9314-ffef63197cb8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:34.802082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446942c3-e69a-4747-9314-ffef63197cb8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:34.803598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446942c3-e69a-4747-9314-ffef63197cb8 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:34.818549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 469/955] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:34.824538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:34.886047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a04e4-f279-42ee-9c8e-15bb3cb2dc57 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.886330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a04e4-f279-42ee-9c8e-15bb3cb2dc57 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.886607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a04e4-f279-42ee-9c8e-15bb3cb2dc57 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.888278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 470/956] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:34.894521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a2043-885f-4958-922f-735023cac138 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.894800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a2043-885f-4958-922f-735023cac138 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.895074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a2043-885f-4958-922f-735023cac138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.914595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-411a2043-885f-4958-922f-735023cac138 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:34.916131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-411a2043-885f-4958-922f-735023cac138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:34.946543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebedd021-f53f-44ae-8cf3-45d5480671cd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:34.960084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:34.983037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:34.983672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:34.983990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:34.984263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:34.992455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:34.993899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:34.999839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 471/957] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:34 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:35.001246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486b300-1329-4c4b-8ab1-bf342911fad6 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.001595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486b300-1329-4c4b-8ab1-bf342911fad6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.002021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486b300-1329-4c4b-8ab1-bf342911fad6 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.003632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 472/958] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:35.009239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca07bd21-305c-4856-b9c9-625e9e90ac7f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.009454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca07bd21-305c-4856-b9c9-625e9e90ac7f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.009744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca07bd21-305c-4856-b9c9-625e9e90ac7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:35.014293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:35.015312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 15 13:57:35.016330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:35.023459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 487/959] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 13:57:34 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:35.024665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.029655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ca07bd21-305c-4856-b9c9-625e9e90ac7f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:35.030836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ca07bd21-305c-4856-b9c9-625e9e90ac7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:35.115515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 473/960] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:35.144332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcd521a-2025-44bf-8ab3-924ef8627673 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:35.180735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:35.181718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.182204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.182694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.183640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:35.184200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:35.185826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:35.204420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 488/961] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:35.215766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca07bd21-305c-4856-b9c9-625e9e90ac7f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.260263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:35.271987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.304642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:35.337717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca07bd21-305c-4856-b9c9-625e9e90ac7f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:35.380082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8501e8d3-a53e-453f-8666-0b08a37d7e99 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:35.384082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1f6df-4d95-4be8-992c-53a2a3bc714f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:35.384714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1f6df-4d95-4be8-992c-53a2a3bc714f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.384996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1f6df-4d95-4be8-992c-53a2a3bc714f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.385274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1f6df-4d95-4be8-992c-53a2a3bc714f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.419969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:35.420607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.421151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.421151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.421731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:35.423129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:35.433360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:35.434440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:35.435328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:35.439462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 489/962] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/services/compute => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:35.440578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.469038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d1f6df-4d95-4be8-992c-53a2a3bc714f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:35.471042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d1f6df-4d95-4be8-992c-53a2a3bc714f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:35.515180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:35.539963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:35.548108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00508c64-5751-4806-b5ab-5bad2fc6b1d5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:35.584029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 474/963] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:57:35.584910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1f6df-4d95-4be8-992c-53a2a3bc714f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.594489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:35.595167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.595518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.596417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.597015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:35.598709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:35.608135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:35.609172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:35.610033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 490/964] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/services/object-store => generated 93 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:35.611414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.697876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1f6df-4d95-4be8-992c-53a2a3bc714f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:35.716885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eda8e5-833a-4c4c-947c-6b9f7c2aaee5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:35.732297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:35.732895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.733175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.733438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.734061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:35.734364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:35.735726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:35.740766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 475/965] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:35.741453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.750358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:35.759930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:35.760567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.760849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.761124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.772425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:35.773809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:35.789792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:35.790756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:35.796544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:35.796544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:35.796544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:35.796544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 491/966] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:35.796544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.809129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:35.855653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e368eb2-b60b-4c97-9c9e-04e24ffb5ac5 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:35.894869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:35.895483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.896927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.897206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.897843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:35.898138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:35.899490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:35.904916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 476/967] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:35.905293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e689727-4052-4d3b-9207-90ef5ce02052 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:35.905685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:35.946413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:35.947024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:35.947301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:35.947572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:35.948226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:35.948526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:35.950283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:35.954553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 492/968] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:35.955435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:36.002014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:36.044358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:36.053366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9dafc-fcea-4b4d-b5ee-f1559afc58e7 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:36.083498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408331b2-3f3d-47b6-abd4-8e00efd493b1 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:36.092845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:36.093457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:36.093734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:36.094003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:36.094647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:36.094949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:36.096898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:36.110226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 477/969] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:36.116706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:36.136439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:36.137067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:36.137329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:36.137667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:36.138306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:36.138602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:36.140004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:36.146821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 493/970] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:35 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:36.152179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:36.259436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c6c53e-8bdf-4260-9dbc-65f1db0410ee None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:36.265885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7653345b-1005-408b-8817-5ad878d5b6b1 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:36.292362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:36.301421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:36.302590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:36.303049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:36.303490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:36.306467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6ecb10-b313-44ae-b75b-34a9b7fe6ae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:36.307182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ecb10-b313-44ae-b75b-34a9b7fe6ae3 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:36.307538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ecb10-b313-44ae-b75b-34a9b7fe6ae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:36.307925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ecb10-b313-44ae-b75b-34a9b7fe6ae3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:36.308682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6ecb10-b313-44ae-b75b-34a9b7fe6ae3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:36.310132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6ecb10-b313-44ae-b75b-34a9b7fe6ae3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:36.312346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:36.314720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:36.320897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:36.330005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:36.331243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:36.332191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:36.332191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:36.332191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:36.332191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:36.332191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:36.332191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:36.332191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:36.332191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:36.339322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 478/971] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:36.340601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:36.341611 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 494/972] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 13:57:36 2022] POST /identity/v3/endpoints => generated 349 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:57:36.458853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ea1f48-b866-47ac-81f1-1d3d46ad6d5b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:36.506554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3137168e-0c97-47e4-bc77-1c84443f3e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:36.507567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3137168e-0c97-47e4-bc77-1c84443f3e80 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:36.508088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3137168e-0c97-47e4-bc77-1c84443f3e80 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:36.508553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3137168e-0c97-47e4-bc77-1c84443f3e80 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:36.509445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3137168e-0c97-47e4-bc77-1c84443f3e80 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:36.509943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3137168e-0c97-47e4-bc77-1c84443f3e80 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:36.511567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3137168e-0c97-47e4-bc77-1c84443f3e80 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:36.525229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 479/973] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:36.531331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6ecb10-b313-44ae-b75b-34a9b7fe6ae3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:36.640868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6ecb10-b313-44ae-b75b-34a9b7fe6ae3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:36.680947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe44a57-7906-4a98-926b-57da38268e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:36.681919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe44a57-7906-4a98-926b-57da38268e92 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:36.682416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe44a57-7906-4a98-926b-57da38268e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:36.682880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe44a57-7906-4a98-926b-57da38268e92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:36.683827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe44a57-7906-4a98-926b-57da38268e92 None admin] RBAC: Adding query filter params (role.id=d511c5452170401cba1f1162075b62be, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=75c8381b6ad84cac881db621d2d99af7) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:36.684433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe44a57-7906-4a98-926b-57da38268e92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:36.686080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe44a57-7906-4a98-926b-57da38268e92 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:36.697480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 495/974] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:36 2022] GET /identity/v3/role_assignments?user.id=75c8381b6ad84cac881db621d2d99af7&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=d511c5452170401cba1f1162075b62be => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:36.904619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6940e0c1-3651-469b-b883-760b5322829f None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:36.905123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6940e0c1-3651-469b-b883-760b5322829f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:36.905123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6940e0c1-3651-469b-b883-760b5322829f None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:36.906779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 480/975] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:36.914656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12927f44-da62-4a05-a811-ca592d9d933d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:36.915114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12927f44-da62-4a05-a811-ca592d9d933d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:36.915578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12927f44-da62-4a05-a811-ca592d9d933d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:36.937445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-12927f44-da62-4a05-a811-ca592d9d933d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:36.938644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-12927f44-da62-4a05-a811-ca592d9d933d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:37.004020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:37.021695 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 496/976] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:36 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:37.303163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6940e0c1-3651-469b-b883-760b5322829f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:37.358985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:37.421771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6940e0c1-3651-469b-b883-760b5322829f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:37.459784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:37.460843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:37.461338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:37.461832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:37.462681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:37.464399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:37.482226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.sql.core [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 15 13:57:37.482226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 15 13:57:37.482226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [parameters: {'user_id': 'ea04f1f482bc4bb3a2b446cd15c8760e', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 15 13:57:37.482226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81654) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result = self._query(query) Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application conn.query(q) Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result.read() Sep 15 13:57:37.490897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 15 13:57:37.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application next(self.gen) Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application yield resource Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application next(self.gen) Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.session.rollback() Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.force_reraise() Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise self.value Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application session.commit() Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 15 13:57:37.493049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._prepare_impl() Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.session.flush() Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._flush(objects) Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application compat.raise_( Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application flush_context.execute() Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rec.execute(self) Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 15 13:57:37.493986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 15 13:57:37.494854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result = self._query(query) Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application conn.query(q) Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application result.read() Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 15 13:57:37.496823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [parameters: {'user_id': 'ea04f1f482bc4bb3a2b446cd15c8760e', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:37.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 15 13:57:37.501088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 15 13:57:37.503389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 15 13:57:37.503389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 15 13:57:37.503389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 15 13:57:37.503389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 15 13:57:37.503389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 15 13:57:37.503389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:37.506631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 481/977] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 15 13:57:37 2022] POST /identity/v3/users => generated 160 bytes in 204 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:37.513728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12927f44-da62-4a05-a811-ca592d9d933d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:37.623545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12927f44-da62-4a05-a811-ca592d9d933d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:37.662011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e15a841-db26-45ed-88c1-f7ba908c5a89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:37.663441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15a841-db26-45ed-88c1-f7ba908c5a89 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:37.663959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15a841-db26-45ed-88c1-f7ba908c5a89 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:37.664439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15a841-db26-45ed-88c1-f7ba908c5a89 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:37.677608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e15a841-db26-45ed-88c1-f7ba908c5a89 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:37.679077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e15a841-db26-45ed-88c1-f7ba908c5a89 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-6e15a841-db26-45ed-88c1-f7ba908c5a89 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:37.693408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:37.695475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:37.696891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 497/978] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:37 2022] GET /identity/v3/users/cinder => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:37.705072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:37.825422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4d91d6-6a19-4935-b27b-25eaa4049739 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:37.836538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:37.865512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa109cc-1952-406c-a2c2-5647b21369a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:37.866482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa109cc-1952-406c-a2c2-5647b21369a2 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:37.866992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa109cc-1952-406c-a2c2-5647b21369a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:37.867473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa109cc-1952-406c-a2c2-5647b21369a2 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:37.868407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa109cc-1952-406c-a2c2-5647b21369a2 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:37.868913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa109cc-1952-406c-a2c2-5647b21369a2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:37.870819 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa109cc-1952-406c-a2c2-5647b21369a2 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:37.916124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 482/979] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 15 13:57:37 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:38.264931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e5275c-03c6-4bfe-9b2b-ddac22ef574c None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:38.265272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e5275c-03c6-4bfe-9b2b-ddac22ef574c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:38.265507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e5275c-03c6-4bfe-9b2b-ddac22ef574c None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:38.267004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 498/980] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:38.272345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe7aa5-276e-4b20-9cdb-4d1f0e380dbf None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:38.272632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe7aa5-276e-4b20-9cdb-4d1f0e380dbf None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:38.272905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe7aa5-276e-4b20-9cdb-4d1f0e380dbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:38.291275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-33fe7aa5-276e-4b20-9cdb-4d1f0e380dbf None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:38.292793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-33fe7aa5-276e-4b20-9cdb-4d1f0e380dbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:38.365990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 483/981] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:38.636870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e5275c-03c6-4bfe-9b2b-ddac22ef574c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:38.690161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:38.748745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e5275c-03c6-4bfe-9b2b-ddac22ef574c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:38.784249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550e17ca-e0ec-4309-8c2a-8ad6e644f22a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:38.784867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550e17ca-e0ec-4309-8c2a-8ad6e644f22a None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:38.785146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550e17ca-e0ec-4309-8c2a-8ad6e644f22a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:38.785417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550e17ca-e0ec-4309-8c2a-8ad6e644f22a None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:38.786081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550e17ca-e0ec-4309-8c2a-8ad6e644f22a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:38.787444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550e17ca-e0ec-4309-8c2a-8ad6e644f22a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:38.804339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 499/982] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:57:38 2022] POST /identity/v3/services => generated 242 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:57:38.855531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96262b79-5bd4-49b1-ad5d-e2265f2e48b3 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:38.855918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96262b79-5bd4-49b1-ad5d-e2265f2e48b3 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:38.856165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96262b79-5bd4-49b1-ad5d-e2265f2e48b3 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:38.857588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 484/983] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:38.864546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc28d8e-906b-4d7a-8ed8-622709c17c92 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:38.864899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc28d8e-906b-4d7a-8ed8-622709c17c92 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:38.865105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc28d8e-906b-4d7a-8ed8-622709c17c92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:38.886148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1dc28d8e-906b-4d7a-8ed8-622709c17c92 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:38.887420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1dc28d8e-906b-4d7a-8ed8-622709c17c92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:38.974192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 500/984] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:38 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:39.113646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4cbae-3b5e-4ba9-a642-32a6bfd01f1c None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.114183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4cbae-3b5e-4ba9-a642-32a6bfd01f1c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.114183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4cbae-3b5e-4ba9-a642-32a6bfd01f1c None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.115664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 485/985] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:39.120692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a6fd5b-c587-43c2-8e78-97239b9f1d1d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.120985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a6fd5b-c587-43c2-8e78-97239b9f1d1d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.121263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a6fd5b-c587-43c2-8e78-97239b9f1d1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.139904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-34a6fd5b-c587-43c2-8e78-97239b9f1d1d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:39.140881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-34a6fd5b-c587-43c2-8e78-97239b9f1d1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:39.168885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:39.212771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 501/986] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:39 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:39.248907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a4cbae-3b5e-4ba9-a642-32a6bfd01f1c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:39.355696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:39.363912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a4cbae-3b5e-4ba9-a642-32a6bfd01f1c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:39.398094 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:39.398637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.398911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.399177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.399775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:39.401119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:39.410633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:39.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:39.413103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 486/987] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:39 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:39.417559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a6fd5b-c587-43c2-8e78-97239b9f1d1d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:39.481925 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:39.524312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a6fd5b-c587-43c2-8e78-97239b9f1d1d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:39.560585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:39.561212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.561489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.561764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.562406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:39.562712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:39.564394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:39.569429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 502/988] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:39 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:39.574844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:39.596959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e54a13-8ad0-4b69-a158-fc225f99c842 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:39.603521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:39.633886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:39.634862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.635384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.635935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] PATH_INFO: `/v3/services/image` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.636736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:39.638191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:39.647254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:39.648536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:39.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 487/989] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Sep 15 13:57:39 2022] GET /identity/v3/services/image => generated 86 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:39.654127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:39.681102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:39.683031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8fc3b0-657e-4d2f-a5e9-1a2d35ab7bcf None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:39.718531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:39.719198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.719578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.719825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.721121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:39.721601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:39.723040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:39.728778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 503/990] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:39 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:39.734663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:39.768658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fe42d8-055a-492a-b1a8-3f69906c100a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:39.805413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:39.806360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.806852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.807329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.808330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] RBAC: Adding query filter params (name=image) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:39.808714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:39.810079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:39.815086 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 488/991] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:39 2022] GET /identity/v3/services?name=image => generated 122 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:39.820720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:39.830821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:39.841426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1926f3ab-38c9-4b50-8cd2-da14d0da2f42 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:39.882614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:39.883541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.883849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.884397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.885111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] RBAC: Adding query filter params (interface=admin, service_id=37c34b7b61b240df8c9ea53d8475cb2e, region_id=RegionOne) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:39.885580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:39.887541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:39.895833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 504/992] 10.222.0.27 () {56 vars in 1249 bytes} [Thu Sep 15 13:57:39 2022] GET /identity/v3/endpoints?service_id=37c34b7b61b240df8c9ea53d8475cb2e&interface=admin®ion_id=RegionOne => generated 193 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:39.899706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:39.920822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:39.938953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f3025d-4a75-4132-b281-5b618a6d6414 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:39.980232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa8bbcf-e45e-414a-bb88-7b40557ad4aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:39.981175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa8bbcf-e45e-414a-bb88-7b40557ad4aa None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:39.981644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa8bbcf-e45e-414a-bb88-7b40557ad4aa None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:39.982090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa8bbcf-e45e-414a-bb88-7b40557ad4aa None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:39.982889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa8bbcf-e45e-414a-bb88-7b40557ad4aa None admin] RBAC: Adding query filter params (type=image) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:39.983360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa8bbcf-e45e-414a-bb88-7b40557ad4aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:39.984928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa8bbcf-e45e-414a-bb88-7b40557ad4aa None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:39.999900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 489/993] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:39 2022] GET /identity/v3/services?type=image => generated 122 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:40.007634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ca5618-0b73-4a41-b7a9-61f409a5e563 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:40.046046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4d2cda-8cd1-4479-b933-f9eb82516b4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:40.046675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4d2cda-8cd1-4479-b933-f9eb82516b4d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:40.046953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4d2cda-8cd1-4479-b933-f9eb82516b4d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:40.047227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4d2cda-8cd1-4479-b933-f9eb82516b4d None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:40.047883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4d2cda-8cd1-4479-b933-f9eb82516b4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:40.049915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4d2cda-8cd1-4479-b933-f9eb82516b4d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:40.064107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 505/994] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 13:57:39 2022] GET /identity/v3/services => generated 1727 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 13:57:41.028250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ee48a-879e-4371-8867-3dfff955aa44 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:41.028812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ee48a-879e-4371-8867-3dfff955aa44 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:41.028812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ee48a-879e-4371-8867-3dfff955aa44 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:41.030275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 490/995] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:41.036018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a88d94-eb01-44f7-b847-05d7c709b4b1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:41.036398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a88d94-eb01-44f7-b847-05d7c709b4b1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:41.036576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a88d94-eb01-44f7-b847-05d7c709b4b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:41.056001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e9a88d94-eb01-44f7-b847-05d7c709b4b1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:41.057185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e9a88d94-eb01-44f7-b847-05d7c709b4b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:41.071156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:41.124045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 506/996] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:41 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:41.396376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ee48a-879e-4371-8867-3dfff955aa44 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:41.498136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ee48a-879e-4371-8867-3dfff955aa44 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:41.532121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:41.532796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:41.533077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:41.533347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:41.533923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:41.535278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:41.543958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:41.544957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:41.545806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:41.548018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 491/997] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:41 2022] GET /identity/v3/services/compute => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:41.552641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a88d94-eb01-44f7-b847-05d7c709b4b1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:41.653047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a88d94-eb01-44f7-b847-05d7c709b4b1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:41.687415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:41.688055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:41.688340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:41.688615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:41.689770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:41.690076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:41.691452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:41.696488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 507/998] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:41 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:41.702084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:41.721281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:41.805432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97460ce-8ad3-4d92-9e4f-d4a029b13bb4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:41.838900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f724ef23-834c-499f-89af-0772568daefb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:41.839512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f724ef23-834c-499f-89af-0772568daefb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:41.839822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f724ef23-834c-499f-89af-0772568daefb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:41.840101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f724ef23-834c-499f-89af-0772568daefb None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:41.840730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f724ef23-834c-499f-89af-0772568daefb None admin] RBAC: Adding query filter params (type=compute) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:41.841035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f724ef23-834c-499f-89af-0772568daefb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:41.842389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f724ef23-834c-499f-89af-0772568daefb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:41.848023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 492/999] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:41 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:41.854104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:41.869994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:41.960279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bad364b-ebe5-4c2a-91ff-312654458008 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:41.995133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351cc313-133e-499d-8551-d9b2c257bff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:41.995780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351cc313-133e-499d-8551-d9b2c257bff3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:41.996066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351cc313-133e-499d-8551-d9b2c257bff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:41.996341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351cc313-133e-499d-8551-d9b2c257bff3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:41.997039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351cc313-133e-499d-8551-d9b2c257bff3 None admin] RBAC: Adding query filter params (interface=public, service_id=12d89812d1b34d3c86821353ebe92478, region_id=RegionOne) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:41.997352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351cc313-133e-499d-8551-d9b2c257bff3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:41.998720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351cc313-133e-499d-8551-d9b2c257bff3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:42.008964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 508/1000] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 15 13:57:41 2022] GET /identity/v3/endpoints?service_id=12d89812d1b34d3c86821353ebe92478&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:42.010193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f724ef23-834c-499f-89af-0772568daefb None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:42.099034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:42.120867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f724ef23-834c-499f-89af-0772568daefb None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:42.158517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90651e1f-f244-4970-9382-7999d941a1f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:42.159132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90651e1f-f244-4970-9382-7999d941a1f2 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:42.159412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90651e1f-f244-4970-9382-7999d941a1f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:42.159709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90651e1f-f244-4970-9382-7999d941a1f2 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:42.160348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90651e1f-f244-4970-9382-7999d941a1f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:42.161700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90651e1f-f244-4970-9382-7999d941a1f2 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:42.173769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 493/1001] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 13:57:42 2022] GET /identity/v3/services => generated 1727 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 13:57:42.182041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb01931-3f61-4adc-85df-c9d7ba802ae5 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:42.182597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb01931-3f61-4adc-85df-c9d7ba802ae5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:42.182597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb01931-3f61-4adc-85df-c9d7ba802ae5 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:42.184037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 509/1002] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:42.190528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cba4c-fb81-4004-862a-892634e2bc60 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:42.190805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cba4c-fb81-4004-862a-892634e2bc60 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:42.191077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cba4c-fb81-4004-862a-892634e2bc60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:42.210743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e31cba4c-fb81-4004-862a-892634e2bc60 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:42.211893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e31cba4c-fb81-4004-862a-892634e2bc60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:42.248320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378eea8d-7748-4858-a181-668bfa1bb7c6 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:42.248640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378eea8d-7748-4858-a181-668bfa1bb7c6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:42.248877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378eea8d-7748-4858-a181-668bfa1bb7c6 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:42.250386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 510/1003] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:42.256556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2a0be7-58b4-430a-8f75-ccc5a03e69c1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:42.256864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2a0be7-58b4-430a-8f75-ccc5a03e69c1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:42.257116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2a0be7-58b4-430a-8f75-ccc5a03e69c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:42.275940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-4d2a0be7-58b4-430a-8f75-ccc5a03e69c1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:42.277168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4d2a0be7-58b4-430a-8f75-ccc5a03e69c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:42.290735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 494/1004] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:42 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:42.364548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 511/1005] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:42 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:42.561125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cba4c-fb81-4004-862a-892634e2bc60 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:42.631307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2a0be7-58b4-430a-8f75-ccc5a03e69c1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:42.639233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:42.641422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:42.667779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cba4c-fb81-4004-862a-892634e2bc60 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:42.706996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a026ce9-967b-44be-ac00-bc47ed0beaa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:42.707617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a026ce9-967b-44be-ac00-bc47ed0beaa1 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:42.707942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a026ce9-967b-44be-ac00-bc47ed0beaa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:42.708218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a026ce9-967b-44be-ac00-bc47ed0beaa1 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:42.708876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a026ce9-967b-44be-ac00-bc47ed0beaa1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:42.710253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a026ce9-967b-44be-ac00-bc47ed0beaa1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:42.727915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 495/1006] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:57:42 2022] POST /identity/v3/services => generated 242 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:42.737303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2a0be7-58b4-430a-8f75-ccc5a03e69c1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:42.783288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:42.783989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:42.784261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:42.784530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:42.785133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:42.786553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:42.796265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:42.797444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 15 13:57:42.798280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:42.802929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 512/1007] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 15 13:57:42 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:42.804257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a026ce9-967b-44be-ac00-bc47ed0beaa1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:42.919477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a026ce9-967b-44be-ac00-bc47ed0beaa1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:42.958273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:42.958888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:42.959167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:42.959437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:42.960106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:42.960412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:42.961797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:42.962901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:42.972268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 496/1008] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:42 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:42.974676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:43.042903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:43.077439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef23a1c9-7027-444d-ab8b-5bd039a2ebee None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:43.115142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc88256-0c94-42ff-841b-b09a8c3b8ba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:43.115779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc88256-0c94-42ff-841b-b09a8c3b8ba2 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:43.116063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc88256-0c94-42ff-841b-b09a8c3b8ba2 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:43.116337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc88256-0c94-42ff-841b-b09a8c3b8ba2 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:43.116976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc88256-0c94-42ff-841b-b09a8c3b8ba2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:43.117281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc88256-0c94-42ff-841b-b09a8c3b8ba2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:43.118654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc88256-0c94-42ff-841b-b09a8c3b8ba2 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:43.129190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 513/1009] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 15 13:57:42 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:43.130340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:43.228744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7c425-ada2-47d6-9c58-a4f9d7205826 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:43.264228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7d81bb-3034-473f-8efa-e83e5f1e30fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:43.265242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7d81bb-3034-473f-8efa-e83e5f1e30fd None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:43.265725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7d81bb-3034-473f-8efa-e83e5f1e30fd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:43.266140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7d81bb-3034-473f-8efa-e83e5f1e30fd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:43.267584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7d81bb-3034-473f-8efa-e83e5f1e30fd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:43.269422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7d81bb-3034-473f-8efa-e83e5f1e30fd None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:43.291735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 497/1010] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 13:57:43 2022] POST /identity/v3/endpoints => generated 325 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:44.398280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a3477-79d3-4626-b99f-8da3fa9fcb8e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:44.398822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a3477-79d3-4626-b99f-8da3fa9fcb8e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:44.398822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a3477-79d3-4626-b99f-8da3fa9fcb8e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:44.400335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 514/1011] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:44.405467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f0f0e-df17-46de-aaff-790b90eab635 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:44.405841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f0f0e-df17-46de-aaff-790b90eab635 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:44.406033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f0f0e-df17-46de-aaff-790b90eab635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:44.424860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-350f0f0e-df17-46de-aaff-790b90eab635 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:44.426040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-350f0f0e-df17-46de-aaff-790b90eab635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:44.481497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:44.503082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 498/1012] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:44 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:44.775400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a3477-79d3-4626-b99f-8da3fa9fcb8e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:44.887568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a3477-79d3-4626-b99f-8da3fa9fcb8e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:44.917204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:44.924302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb656b7-31f3-43ae-b859-1884d282ae53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:44.924949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb656b7-31f3-43ae-b859-1884d282ae53 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:44.925233 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb656b7-31f3-43ae-b859-1884d282ae53 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:44.925509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb656b7-31f3-43ae-b859-1884d282ae53 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:44.926100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb656b7-31f3-43ae-b859-1884d282ae53 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:44.927822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb656b7-31f3-43ae-b859-1884d282ae53 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-cdb656b7-31f3-43ae-b859-1884d282ae53 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:44.936997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:44.937991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 15 13:57:44.939509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:44.943652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 515/1013] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:44 2022] GET /identity/v3/services/compute => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:44.945007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f0f0e-df17-46de-aaff-790b90eab635 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:44.952071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de68012a-52e6-44d2-97f5-8bc734ba32d6 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:44.952359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de68012a-52e6-44d2-97f5-8bc734ba32d6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:44.952631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de68012a-52e6-44d2-97f5-8bc734ba32d6 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:44.954085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 516/1014] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:57:44.959837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f2066b-f114-46d4-99b4-63693ebb8fc9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:44.960120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f2066b-f114-46d4-99b4-63693ebb8fc9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:44.960393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f2066b-f114-46d4-99b4-63693ebb8fc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:44.978428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-33f2066b-f114-46d4-99b4-63693ebb8fc9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:44.979622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-33f2066b-f114-46d4-99b4-63693ebb8fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:45.054079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f0f0e-df17-46de-aaff-790b90eab635 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:45.061367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 517/1015] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:44 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:45.092079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:45.092996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.093459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.093889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.094665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:45.095179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:45.097514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:45.103730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 499/1016] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:44 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:45.108293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f2066b-f114-46d4-99b4-63693ebb8fc9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:45.210822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f2066b-f114-46d4-99b4-63693ebb8fc9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:45.237256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:45.246667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:45.247280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.247567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.247883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.248523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] RBAC: Adding query filter params (type=compute) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:45.248834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:45.250206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:45.255843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 518/1017] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 15 13:57:45 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:45.261020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:45.321822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:45.328954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:45.384386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e9d477-bf9e-44f6-a7c8-1699adfbe2f8 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:45.428328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a1080f-a23b-45b1-9413-d788b8b05a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:45.429377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1080f-a23b-45b1-9413-d788b8b05a57 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.429665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1080f-a23b-45b1-9413-d788b8b05a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.429992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1080f-a23b-45b1-9413-d788b8b05a57 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.430627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a1080f-a23b-45b1-9413-d788b8b05a57 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:45.432034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a1080f-a23b-45b1-9413-d788b8b05a57 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:45.440345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc699a86-71ef-4d47-b902-d34b1400c68a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:45.453856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 500/1018] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 13:57:45 2022] POST /identity/v3/endpoints => generated 334 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:57:45.456571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e1d7bf-cc91-46c4-868c-cd8398369b36 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.456853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e1d7bf-cc91-46c4-868c-cd8398369b36 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.457121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e1d7bf-cc91-46c4-868c-cd8398369b36 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.458389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 501/1019] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:45.464140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c24bd6-6ce0-48fb-b976-6fe1aede7f27 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.464460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c24bd6-6ce0-48fb-b976-6fe1aede7f27 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.464695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c24bd6-6ce0-48fb-b976-6fe1aede7f27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.485989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f5c24bd6-6ce0-48fb-b976-6fe1aede7f27 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:45.487210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f5c24bd6-6ce0-48fb-b976-6fe1aede7f27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:45.494912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:45.495569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.495890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.496163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] PATH_INFO: `/v3/services/image` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.496760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:45.498164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:45.508454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:45.509534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:45.510434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 519/1020] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Sep 15 13:57:45 2022] GET /identity/v3/services/image => generated 86 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:45.515569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:45.573919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:45.577427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 502/1021] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:45 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:45.634783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a87aab-d1e4-48ef-886d-fe7ec47c41eb None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:45.671640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:45.672507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.672792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.673062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.673719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] RBAC: Adding query filter params (name=image) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:45.674019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:45.675409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:45.680889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 520/1022] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:45 2022] GET /identity/v3/services?name=image => generated 122 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:45.686600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c24bd6-6ce0-48fb-b976-6fe1aede7f27 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:45.722568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:45.792092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c24bd6-6ce0-48fb-b976-6fe1aede7f27 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:45.826842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:45.827368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.827642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.827939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.828564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] RBAC: Adding query filter params (type=image) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:45.828861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:45.830271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:45.835201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 503/1023] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:45 2022] GET /identity/v3/services?type=image => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:45.840871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:45.851859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:45.950607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a08b7e-69e5-43d5-8908-486634135f21 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:45.960410 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:45.964766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470884fa-d595-4c75-986d-1ef96a936680 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:45.966944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:45.992702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:45.993462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:45.993861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:45.994229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:45.995037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] RBAC: Adding query filter params (interface=public, service_id=30288b95d34f433a8bc0d54066d78e07, region_id=RegionOne) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:45.995425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:45.996831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:46.002236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 521/1024] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 15 13:57:45 2022] GET /identity/v3/endpoints?service_id=30288b95d34f433a8bc0d54066d78e07&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:46.003572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60887c0-3219-4a0b-ab6a-b3123ab697d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:46.004420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60887c0-3219-4a0b-ab6a-b3123ab697d9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:46.004849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60887c0-3219-4a0b-ab6a-b3123ab697d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:46.005269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60887c0-3219-4a0b-ab6a-b3123ab697d9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:46.006524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60887c0-3219-4a0b-ab6a-b3123ab697d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:46.007886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:46.008391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60887c0-3219-4a0b-ab6a-b3123ab697d9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:46.010475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 504/1025] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:46.015800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60887c0-3219-4a0b-ab6a-b3123ab697d9 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:46.118760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c531fc-48a7-4af4-8af5-ac276dc4e467 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:46.130852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60887c0-3219-4a0b-ab6a-b3123ab697d9 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:46.153847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0701e321-8d23-45fb-8186-b7b10f5f402f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:46.154457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0701e321-8d23-45fb-8186-b7b10f5f402f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:46.154740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0701e321-8d23-45fb-8186-b7b10f5f402f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:46.155012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0701e321-8d23-45fb-8186-b7b10f5f402f None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:46.155618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0701e321-8d23-45fb-8186-b7b10f5f402f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:46.157053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0701e321-8d23-45fb-8186-b7b10f5f402f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:46.164847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 522/1026] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 13:57:46 2022] GET /identity/v3/services => generated 1957 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 13:57:46.166554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20694f4f-683c-413b-98e9-fd50133cb913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:46.167171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20694f4f-683c-413b-98e9-fd50133cb913 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:46.167455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20694f4f-683c-413b-98e9-fd50133cb913 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:46.167763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20694f4f-683c-413b-98e9-fd50133cb913 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:46.168425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20694f4f-683c-413b-98e9-fd50133cb913 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:46.169886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20694f4f-683c-413b-98e9-fd50133cb913 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:46.201204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 505/1027] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:57:46 2022] POST /identity/v3/projects => generated 311 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:47.796444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26a67f0-5de1-4b10-8387-6f1185f7fbd6 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:47.796988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26a67f0-5de1-4b10-8387-6f1185f7fbd6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:47.796988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26a67f0-5de1-4b10-8387-6f1185f7fbd6 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:47.798608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 523/1028] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:47.804499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c94be-9004-4506-a63b-c87884bdfe3f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:47.804944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c94be-9004-4506-a63b-c87884bdfe3f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:47.805118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c94be-9004-4506-a63b-c87884bdfe3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:47.823833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6e2c94be-9004-4506-a63b-c87884bdfe3f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:47.825012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6e2c94be-9004-4506-a63b-c87884bdfe3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:47.840046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:47.897562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 506/1029] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:47 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:48.176209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26a67f0-5de1-4b10-8387-6f1185f7fbd6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:48.222966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:48.284322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26a67f0-5de1-4b10-8387-6f1185f7fbd6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:48.327491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d328d1b-d059-4ca0-898a-c2b0c5019b96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:48.328197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d328d1b-d059-4ca0-898a-c2b0c5019b96 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.328472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d328d1b-d059-4ca0-898a-c2b0c5019b96 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.328741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d328d1b-d059-4ca0-898a-c2b0c5019b96 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.342097 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d328d1b-d059-4ca0-898a-c2b0c5019b96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:48.344509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09883455-f1f6-41e1-8586-aa144339be90 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.344793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09883455-f1f6-41e1-8586-aa144339be90 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.345681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09883455-f1f6-41e1-8586-aa144339be90 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.347431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 507/1030] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3d328d1b-d059-4ca0-898a-c2b0c5019b96 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:48.347536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:48.348486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:48.349351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:48.351973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 524/1031] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 15 13:57:48 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:48.352478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de646216-5264-45ea-9363-058f1b88d1db None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.352764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de646216-5264-45ea-9363-058f1b88d1db None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.353040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de646216-5264-45ea-9363-058f1b88d1db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.354353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d328d1b-d059-4ca0-898a-c2b0c5019b96 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:48.370984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-de646216-5264-45ea-9363-058f1b88d1db None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:48.372183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-de646216-5264-45ea-9363-058f1b88d1db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:48.447415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 508/1032] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:48 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:48.448590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb0d85-dc7f-4b65-8d32-ca74875e365d None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.448873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb0d85-dc7f-4b65-8d32-ca74875e365d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.449144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb0d85-dc7f-4b65-8d32-ca74875e365d None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.450571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 509/1033] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:48.457205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ad020-a1f7-40c7-879d-9479712855c2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.457494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ad020-a1f7-40c7-879d-9479712855c2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.457781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ad020-a1f7-40c7-879d-9479712855c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.473071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d328d1b-d059-4ca0-898a-c2b0c5019b96 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:48.474754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:48.476256 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-436ad020-a1f7-40c7-879d-9479712855c2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:48.477563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-436ad020-a1f7-40c7-879d-9479712855c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:48.511226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:48.512212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.512716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.513284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.513949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:48.514256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:48.515615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:48.530854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 525/1034] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 15 13:57:48 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:48.531845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:48.555829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 510/1035] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:48 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:48.633021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd754aa-8fcd-4be7-8139-4d857041a64e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:48.666547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:48.667170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.667449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.667744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.678545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:48.680013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:48.695041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:48.701130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:48.702089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:48.702089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:48.702089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:48.702089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 526/1036] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:48.703441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436ad020-a1f7-40c7-879d-9479712855c2 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:48.713539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:48.714772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:48.734378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:48.816147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436ad020-a1f7-40c7-879d-9479712855c2 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:48.823200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591a0b21-110b-4a33-a89d-2daba2c993d4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:48.852677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:48.853285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.853561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.853833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.854479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:48.854776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:48.856182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:48.862304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:48.862930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:48.863212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:48.863483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] PATH_INFO: `/v3/services/image` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:48.864085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:48.865437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:48.867822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 511/1037] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:48.868602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:48.874393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:48.879874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:48.880753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 527/1038] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Sep 15 13:57:48 2022] GET /identity/v3/services/image => generated 86 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:48.881548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:48.937442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:48.962410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:48.973564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04340db-efc6-4286-82c8-4f1329e4ff42 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:48.982347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60caba42-6fb3-4651-81a2-851a93819886 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:49.006108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abbc9f9-44ce-4c60-9f6b-373c44a316d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.006918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abbc9f9-44ce-4c60-9f6b-373c44a316d4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.006918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abbc9f9-44ce-4c60-9f6b-373c44a316d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.007149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abbc9f9-44ce-4c60-9f6b-373c44a316d4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.008285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abbc9f9-44ce-4c60-9f6b-373c44a316d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.009705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abbc9f9-44ce-4c60-9f6b-373c44a316d4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.011270 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 512/1039] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:49.012055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abbc9f9-44ce-4c60-9f6b-373c44a316d4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:49.018299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.018940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.019222 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.019497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.028005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.029455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:49.041473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:49.042704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 528/1040] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:57:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:49.043909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:49.115575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abbc9f9-44ce-4c60-9f6b-373c44a316d4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:49.148500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02bb639-897a-499e-8930-9dcd6b021167 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:49.151781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.152392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.152672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.152945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.153573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] RBAC: Adding query filter params (name=image) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:49.153877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.155237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.156972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:49.161276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 513/1041] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:49 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:49.162063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:49.183397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1738c0f5-cb48-4697-8e22-4997b002743a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.184039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738c0f5-cb48-4697-8e22-4997b002743a None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.184319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738c0f5-cb48-4697-8e22-4997b002743a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.184592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738c0f5-cb48-4697-8e22-4997b002743a None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.185236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1738c0f5-cb48-4697-8e22-4997b002743a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.186665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1738c0f5-cb48-4697-8e22-4997b002743a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.197950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:49.210243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 529/1042] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:57:49 2022] POST /identity/v3/users => generated 285 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:57:49.211044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1738c0f5-cb48-4697-8e22-4997b002743a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:49.284137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa8998e-a9ae-4436-98e3-fcd19ca1eb50 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:49.328966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95373a77-658b-4379-8297-4188360f27cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.331517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95373a77-658b-4379-8297-4188360f27cb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.331834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95373a77-658b-4379-8297-4188360f27cb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.332109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95373a77-658b-4379-8297-4188360f27cb None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.332887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95373a77-658b-4379-8297-4188360f27cb None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:49.333191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95373a77-658b-4379-8297-4188360f27cb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.334645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95373a77-658b-4379-8297-4188360f27cb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.335383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1738c0f5-cb48-4697-8e22-4997b002743a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:49.359577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 514/1043] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:57:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:49.361571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95373a77-658b-4379-8297-4188360f27cb None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:49.377048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.377934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.378369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.378649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.379298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] RBAC: Adding query filter params (type=image) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:49.379600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.381116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.387359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 530/1044] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:49 2022] GET /identity/v3/services?type=image => generated 350 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:49.393163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:49.437357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:49.463278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:49.488900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95373a77-658b-4379-8297-4188360f27cb None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:49.509134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e341a575-adf2-418c-8c72-460f86e3a0c6 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:49.524497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4481762-fe8d-4b5b-9991-4f8fcdaa1770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.525117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4481762-fe8d-4b5b-9991-4f8fcdaa1770 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.525395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4481762-fe8d-4b5b-9991-4f8fcdaa1770 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.525670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4481762-fe8d-4b5b-9991-4f8fcdaa1770 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.546701 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf368d8a-9c2d-4c15-8f18-dc4419267be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.547232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf368d8a-9c2d-4c15-8f18-dc4419267be6 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.547512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf368d8a-9c2d-4c15-8f18-dc4419267be6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.549071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf368d8a-9c2d-4c15-8f18-dc4419267be6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.549919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf368d8a-9c2d-4c15-8f18-dc4419267be6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.551305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf368d8a-9c2d-4c15-8f18-dc4419267be6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.572635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 531/1045] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 13:57:49 2022] POST /identity/v3/endpoints => generated 327 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:57:49.600792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4481762-fe8d-4b5b-9991-4f8fcdaa1770 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.602333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4481762-fe8d-4b5b-9991-4f8fcdaa1770 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.669607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:49.722773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 515/1046] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:49 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:57:49.724489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf368d8a-9c2d-4c15-8f18-dc4419267be6 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:49.840910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf368d8a-9c2d-4c15-8f18-dc4419267be6 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:49.854477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:49.878027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:49.878656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:49.878935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:49.879205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:49.890031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:49.891463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:49.906051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:49.907025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:49.908481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:49.908481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:49.908481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:49.908481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 532/1047] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:49.913634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4481762-fe8d-4b5b-9991-4f8fcdaa1770 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:50.016368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4481762-fe8d-4b5b-9991-4f8fcdaa1770 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:50.051176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:50.051789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:50.052330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:50.052613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:50.053239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:50.053539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:50.054899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:50.071441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 516/1048] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:50.073079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:50.178802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a890357-dfa7-46d4-8a9f-960abda0e71a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:50.212674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:50.213303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:50.213572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:50.213840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:50.220947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:50.222592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:50.233537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:50.234395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:50.240453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 533/1049] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:50.242418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:50.262239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:50.350895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e820162b-a39a-4312-a9fa-aa7e06fb6b1d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:50.385356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9f8aa0-ca4e-4e20-8779-1b7e64ec42ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:50.385957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f8aa0-ca4e-4e20-8779-1b7e64ec42ed None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:50.386233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f8aa0-ca4e-4e20-8779-1b7e64ec42ed None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:50.386501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f8aa0-ca4e-4e20-8779-1b7e64ec42ed None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:50.387183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9f8aa0-ca4e-4e20-8779-1b7e64ec42ed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:50.387486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9f8aa0-ca4e-4e20-8779-1b7e64ec42ed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:50.388936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9f8aa0-ca4e-4e20-8779-1b7e64ec42ed None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:50.406265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 517/1050] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:50.407603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:50.435072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:50.515580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc76957-4339-4fb8-b68e-aa92c6a4d517 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:50.552354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cad1f9e-f02c-42d1-812f-b1825f7a8379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:50.552966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad1f9e-f02c-42d1-812f-b1825f7a8379 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:50.553244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad1f9e-f02c-42d1-812f-b1825f7a8379 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:50.553514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad1f9e-f02c-42d1-812f-b1825f7a8379 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:50.554237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cad1f9e-f02c-42d1-812f-b1825f7a8379 None admin] RBAC: Adding query filter params (role.id=d511c5452170401cba1f1162075b62be, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=ce6744790db74f0aaffc6eb269bf6737) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:50.554546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cad1f9e-f02c-42d1-812f-b1825f7a8379 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:50.555996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cad1f9e-f02c-42d1-812f-b1825f7a8379 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:50.567959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 534/1051] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:50 2022] GET /identity/v3/role_assignments?user.id=ce6744790db74f0aaffc6eb269bf6737&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=d511c5452170401cba1f1162075b62be => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:51.462198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880b760-7f6c-4917-9c0f-11e82a08fdab None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:51.462707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880b760-7f6c-4917-9c0f-11e82a08fdab None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:51.462784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d880b760-7f6c-4917-9c0f-11e82a08fdab None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:51.464130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 518/1052] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:51.469516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd2cc3f-9679-4d4d-b302-5db2f85024da None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:51.470416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd2cc3f-9679-4d4d-b302-5db2f85024da None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:51.470606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd2cc3f-9679-4d4d-b302-5db2f85024da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:51.500323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1cd2cc3f-9679-4d4d-b302-5db2f85024da None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:51.501699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1cd2cc3f-9679-4d4d-b302-5db2f85024da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:51.581592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 535/1053] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:51.801821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0f74d9-5e73-47bb-b5da-c6db4935e544 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:51.802398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0f74d9-5e73-47bb-b5da-c6db4935e544 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:51.802640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0f74d9-5e73-47bb-b5da-c6db4935e544 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:51.804252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 519/1054] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:51.809680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f634e1-dd47-4fa2-9a37-a9e87d246fc5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:51.809978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f634e1-dd47-4fa2-9a37-a9e87d246fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:51.810256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f634e1-dd47-4fa2-9a37-a9e87d246fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:51.830331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-51f634e1-dd47-4fa2-9a37-a9e87d246fc5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:51.831523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-51f634e1-dd47-4fa2-9a37-a9e87d246fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:51.849080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:51.885014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0f74d9-5e73-47bb-b5da-c6db4935e544 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:51.913758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 536/1055] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:51.969925 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:51.993822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0f74d9-5e73-47bb-b5da-c6db4935e544 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:52.028595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b924bfe3-eb4a-4984-beb6-6aec657c3301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:52.029213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b924bfe3-eb4a-4984-beb6-6aec657c3301 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.029488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b924bfe3-eb4a-4984-beb6-6aec657c3301 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.029756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b924bfe3-eb4a-4984-beb6-6aec657c3301 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.040860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b924bfe3-eb4a-4984-beb6-6aec657c3301 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-b924bfe3-eb4a-4984-beb6-6aec657c3301 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:52.045511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:52.046519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:52.048741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 520/1056] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:52.052832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f634e1-dd47-4fa2-9a37-a9e87d246fc5 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:52.153778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f634e1-dd47-4fa2-9a37-a9e87d246fc5 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:52.188382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56442196-6874-4b89-8840-aaf836587088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:52.189600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56442196-6874-4b89-8840-aaf836587088 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.189887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56442196-6874-4b89-8840-aaf836587088 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.190155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56442196-6874-4b89-8840-aaf836587088 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.190815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56442196-6874-4b89-8840-aaf836587088 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:52.191119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56442196-6874-4b89-8840-aaf836587088 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:52.192510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56442196-6874-4b89-8840-aaf836587088 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:52.200856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 537/1057] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:52.206634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b924bfe3-eb4a-4984-beb6-6aec657c3301 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:52.208195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56442196-6874-4b89-8840-aaf836587088 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:52.221542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:52.315150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b924bfe3-eb4a-4984-beb6-6aec657c3301 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:52.320939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56442196-6874-4b89-8840-aaf836587088 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:52.351972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5c5be8-caa0-4c80-b76d-0606bc936f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:52.352609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5c5be8-caa0-4c80-b76d-0606bc936f7f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.352887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5c5be8-caa0-4c80-b76d-0606bc936f7f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.353301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5c5be8-caa0-4c80-b76d-0606bc936f7f None admin] PATH_INFO: `/v3/users/2e00af28daae4a8d8e3eaf0fd7508f76` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.358120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7b2b83-4d97-466e-8683-bbbd1c0a9c7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:52.358732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7b2b83-4d97-466e-8683-bbbd1c0a9c7f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.359012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7b2b83-4d97-466e-8683-bbbd1c0a9c7f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.359286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7b2b83-4d97-466e-8683-bbbd1c0a9c7f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.362634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:52.369064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5c5be8-caa0-4c80-b76d-0606bc936f7f None admin] RBAC: Authorizing `identity:get_user(user_id=2e00af28daae4a8d8e3eaf0fd7508f76)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:52.370729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5c5be8-caa0-4c80-b76d-0606bc936f7f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:52.372604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 521/1058] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:57:52 2022] GET /identity/v3/users/2e00af28daae4a8d8e3eaf0fd7508f76 => generated 285 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:52.380583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5c5be8-caa0-4c80-b76d-0606bc936f7f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:52.435585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7b2b83-4d97-466e-8683-bbbd1c0a9c7f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:52.437001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7b2b83-4d97-466e-8683-bbbd1c0a9c7f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:52.449872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:52.494451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5c5be8-caa0-4c80-b76d-0606bc936f7f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:52.530101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77800c18-2cd6-41ab-9933-be1d89df1b2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:52.530826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77800c18-2cd6-41ab-9933-be1d89df1b2c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.531110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77800c18-2cd6-41ab-9933-be1d89df1b2c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.531500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77800c18-2cd6-41ab-9933-be1d89df1b2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.540894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 538/1059] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 15 13:57:52.546592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7b2b83-4d97-466e-8683-bbbd1c0a9c7f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:52.605118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:52.614801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77800c18-2cd6-41ab-9933-be1d89df1b2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:52.616333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77800c18-2cd6-41ab-9933-be1d89df1b2c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:52.656582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7b2b83-4d97-466e-8683-bbbd1c0a9c7f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:52.669086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:52.696496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3833ff72-64d3-426f-8835-3772399135af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:52.697360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3833ff72-64d3-426f-8835-3772399135af None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.697639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3833ff72-64d3-426f-8835-3772399135af None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.697906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3833ff72-64d3-426f-8835-3772399135af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.709011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3833ff72-64d3-426f-8835-3772399135af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:52.710462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3833ff72-64d3-426f-8835-3772399135af None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:52.725376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 522/1060] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3833ff72-64d3-426f-8835-3772399135af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:52.725893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:52.726818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:52.727761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:52.727761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:52.727761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:52.727935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ce58f6-0478-4b8c-9512-120468cefba0 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.728005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 539/1061] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:52.728085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ce58f6-0478-4b8c-9512-120468cefba0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.728169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ce58f6-0478-4b8c-9512-120468cefba0 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.729509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 523/1062] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:52.731465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3833ff72-64d3-426f-8835-3772399135af None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:52.732618 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ce58f6-0478-4b8c-9512-120468cefba0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:52.826585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:52.832777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3833ff72-64d3-426f-8835-3772399135af None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:52.834060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ce58f6-0478-4b8c-9512-120468cefba0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:52.866831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10ecf1b-d6fa-49fe-a533-723d5b192de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:52.867455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ecf1b-d6fa-49fe-a533-723d5b192de7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.867767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ecf1b-d6fa-49fe-a533-723d5b192de7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.867886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:52.868049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ecf1b-d6fa-49fe-a533-723d5b192de7 None admin] PATH_INFO: `/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.868377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.868655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.868926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.869154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10ecf1b-d6fa-49fe-a533-723d5b192de7 None admin] RBAC: Authorizing `identity:get_project(project_id=cc6f33ffe30f4ef0941a62f2a03e9810)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:52.869553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:52.869853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:52.870602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10ecf1b-d6fa-49fe-a533-723d5b192de7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:52.871185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:52.872259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 540/1063] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:57:52 2022] GET /identity/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:52.873689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98641ca4-84fa-4b64-8716-1b64ae20195f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:52.873689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98641ca4-84fa-4b64-8716-1b64ae20195f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:52.873956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98641ca4-84fa-4b64-8716-1b64ae20195f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:52.887444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 524/1064] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:52.887444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:52.893192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-98641ca4-84fa-4b64-8716-1b64ae20195f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:52.894434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-98641ca4-84fa-4b64-8716-1b64ae20195f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:52.921909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:52.976321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 541/1065] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:53.003275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bbec52-cc06-4425-a18e-fcd10fc3f7f1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:53.041257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:53.041934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:53.042220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:53.042520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:53.043262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=cc6f33ffe30f4ef0941a62f2a03e9810, user.id=2e00af28daae4a8d8e3eaf0fd7508f76) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:53.043577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:53.045073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:53.051533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 525/1066] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:52 2022] GET /identity/v3/role_assignments?user.id=2e00af28daae4a8d8e3eaf0fd7508f76&scope.project.id=cc6f33ffe30f4ef0941a62f2a03e9810&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:53.242721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98641ca4-84fa-4b64-8716-1b64ae20195f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:53.346006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98641ca4-84fa-4b64-8716-1b64ae20195f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:53.379781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46c56cb-9d63-4e85-866b-8c669881375c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:53.380408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c56cb-9d63-4e85-866b-8c669881375c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:53.380715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c56cb-9d63-4e85-866b-8c669881375c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:53.380986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c56cb-9d63-4e85-866b-8c669881375c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:53.390806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46c56cb-9d63-4e85-866b-8c669881375c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-e46c56cb-9d63-4e85-866b-8c669881375c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:53.396201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:53.397134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:53.398122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 542/1067] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 15 13:57:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:53.403266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:53.425642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:53.502166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def00dc4-84ef-4792-ad5e-99c53ab892d6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:53.534758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:53.535361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:53.535639 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:53.535937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:53.536598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:53.536899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:53.538259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:53.547330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 526/1068] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 15 13:57:53 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:53.553836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46c56cb-9d63-4e85-866b-8c669881375c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:53.576102 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:53.651232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46c56cb-9d63-4e85-866b-8c669881375c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:53.684887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:53.685507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:53.685780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:53.686049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:53.696149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:53.697583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:53.711733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:53.712713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:53.714063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:53.714063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:53.714063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:53.714063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 543/1069] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:53.718465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:53.797440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:53.816196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a203179a-ca31-4faa-9760-e69e173c1f89 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:53.848802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:53.849401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:53.849678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:53.849947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:53.850574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:53.850871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:53.852263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:53.863272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 527/1070] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:53.868934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:53.941759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:53.972713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b058b0-b576-4b46-8492-683cd093d6dd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:54.005723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:54.006349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:54.006630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:54.006903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:54.014118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:54.015548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:54.026525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:54.027475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:54.029040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 544/1071] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:57:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:57:54.033606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:54.135642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2912a565-5cf9-43ec-9b33-7ebcbee25742 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:54.167516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:54.168143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:54.168428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:54.168704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:54.169423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:54.169728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:54.171133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:54.185837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:54.190357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 528/1072] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:57:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:54.195910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:54.293405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad278f2d-3361-45dd-a92d-b5e0404169e0 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:54.325599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:54.331316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:54.331989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:54.332273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:54.332549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:54.343010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:54.344538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:54.358350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:54.359514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:54.361143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:54.361143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:54.361143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:54.361143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 545/1073] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:54.365738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:54.463821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff76930-b413-49fe-a5cd-538a5ee117d0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:54.496709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:54.497305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:54.497580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:54.497849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:54.498476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:54.498774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:54.500160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:54.511702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 529/1074] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:54.518077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:54.616019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4ba23-9ce4-4289-a5df-2fa6c1d8bf0a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:54.649042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:54.649662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:54.649942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:54.650210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:54.656709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:54.659041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:54.670000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:54.670897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:54.671816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 546/1075] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:54.677447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:54.736211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:54.776668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f74dcc-d59e-492a-b9d6-6b0a5d1ea3e4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:54.810179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbeb4a8-97db-413f-aade-34a05fcc6ed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:54.810782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeb4a8-97db-413f-aade-34a05fcc6ed3 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:54.811064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeb4a8-97db-413f-aade-34a05fcc6ed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:54.811336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeb4a8-97db-413f-aade-34a05fcc6ed3 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:54.812061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbeb4a8-97db-413f-aade-34a05fcc6ed3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:54.812367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbeb4a8-97db-413f-aade-34a05fcc6ed3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:54.813863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbeb4a8-97db-413f-aade-34a05fcc6ed3 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:54.825739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 530/1076] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:54.832244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:54.878227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:54.935223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd42329-2c49-4ac5-b290-ea337ce88ad4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:54.969220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a932b0ae-e9f9-47ec-9d7e-02ac732b2bf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:54.969944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a932b0ae-e9f9-47ec-9d7e-02ac732b2bf8 None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:54.970881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a932b0ae-e9f9-47ec-9d7e-02ac732b2bf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:54.971167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a932b0ae-e9f9-47ec-9d7e-02ac732b2bf8 None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/ce6744790db74f0aaffc6eb269bf6737/roles/d511c5452170401cba1f1162075b62be` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:54.973308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a932b0ae-e9f9-47ec-9d7e-02ac732b2bf8 None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=ce6744790db74f0aaffc6eb269bf6737, role_id=d511c5452170401cba1f1162075b62be)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:54.974748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a932b0ae-e9f9-47ec-9d7e-02ac732b2bf8 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:54.987360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 547/1077] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:57:54 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/ce6744790db74f0aaffc6eb269bf6737/roles/d511c5452170401cba1f1162075b62be => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:57:55.195810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddb22d-1521-4392-933d-9b3ca71a85c7 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:55.196102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddb22d-1521-4392-933d-9b3ca71a85c7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:55.196384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddb22d-1521-4392-933d-9b3ca71a85c7 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:55.197725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 531/1078] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:55.203017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfe131e-c494-4a93-8891-aa48b84d2c1f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:55.203301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfe131e-c494-4a93-8891-aa48b84d2c1f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:55.203574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfe131e-c494-4a93-8891-aa48b84d2c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:55.222206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-dbfe131e-c494-4a93-8891-aa48b84d2c1f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:55.223361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dbfe131e-c494-4a93-8891-aa48b84d2c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:55.288730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 548/1079] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:55.553580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfddb22d-1521-4392-933d-9b3ca71a85c7 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:55.653131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfddb22d-1521-4392-933d-9b3ca71a85c7 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:55.674484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:55.689331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0686bc-ea3b-49a2-b427-5513ec0dedbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:55.690125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0686bc-ea3b-49a2-b427-5513ec0dedbd None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:55.690401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0686bc-ea3b-49a2-b427-5513ec0dedbd None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:55.690667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0686bc-ea3b-49a2-b427-5513ec0dedbd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:55.700540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0686bc-ea3b-49a2-b427-5513ec0dedbd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-5a0686bc-ea3b-49a2-b427-5513ec0dedbd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:55.705484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:55.706601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:55.707630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:55.714474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 532/1080] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:55.714615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfe131e-c494-4a93-8891-aa48b84d2c1f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:55.746340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:55.812679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfe131e-c494-4a93-8891-aa48b84d2c1f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:55.845800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:55.846429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:55.846704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:55.846970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:55.847649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:55.847979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:55.849398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:55.858594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 549/1081] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:55.865334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0686bc-ea3b-49a2-b427-5513ec0dedbd None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:55.961830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0686bc-ea3b-49a2-b427-5513ec0dedbd None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:55.995102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fb6ff6-eadb-41c7-afc7-072526b2d1a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:55.995732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb6ff6-eadb-41c7-afc7-072526b2d1a4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:55.996008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb6ff6-eadb-41c7-afc7-072526b2d1a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:55.996274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fb6ff6-eadb-41c7-afc7-072526b2d1a4 None admin] PATH_INFO: `/v3/users/2e00af28daae4a8d8e3eaf0fd7508f76` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:55.997359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fb6ff6-eadb-41c7-afc7-072526b2d1a4 None admin] RBAC: Authorizing `identity:get_user(user_id=2e00af28daae4a8d8e3eaf0fd7508f76)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:55.998763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fb6ff6-eadb-41c7-afc7-072526b2d1a4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:56.000329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 533/1082] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:57:55 2022] GET /identity/v3/users/2e00af28daae4a8d8e3eaf0fd7508f76 => generated 285 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:56.006282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:56.098197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:56.104295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfc22eb-1d95-40e9-9e61-7a143575d425 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:56.137206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5bfe0c-aa1c-4dab-955e-92bd6c39695b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:56.137836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5bfe0c-aa1c-4dab-955e-92bd6c39695b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:56.138116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5bfe0c-aa1c-4dab-955e-92bd6c39695b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:56.138394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5bfe0c-aa1c-4dab-955e-92bd6c39695b None admin] PATH_INFO: `/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:56.139533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5bfe0c-aa1c-4dab-955e-92bd6c39695b None admin] RBAC: Authorizing `identity:get_project(project_id=cc6f33ffe30f4ef0941a62f2a03e9810)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:56.140993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5bfe0c-aa1c-4dab-955e-92bd6c39695b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:56.143598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 550/1083] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:57:56 2022] GET /identity/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:56.149228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fb6ff6-eadb-41c7-afc7-072526b2d1a4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:56.197630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:56.247950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fb6ff6-eadb-41c7-afc7-072526b2d1a4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:56.280307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a5b92b-1b81-43ef-9518-364e38797273 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:56.280958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5b92b-1b81-43ef-9518-364e38797273 None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:56.281235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5b92b-1b81-43ef-9518-364e38797273 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:56.281506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5b92b-1b81-43ef-9518-364e38797273 None admin] PATH_INFO: `/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810/users/2e00af28daae4a8d8e3eaf0fd7508f76/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:56.284051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a5b92b-1b81-43ef-9518-364e38797273 None admin] RBAC: Authorizing `identity:create_grant(project_id=cc6f33ffe30f4ef0941a62f2a03e9810, user_id=2e00af28daae4a8d8e3eaf0fd7508f76, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:56.285461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a5b92b-1b81-43ef-9518-364e38797273 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:56.298085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 534/1084] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:57:56 2022] PUT /identity/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810/users/2e00af28daae4a8d8e3eaf0fd7508f76/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:57:57.142457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b178fe-939f-47bf-a84f-6839009d8694 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:57.143055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b178fe-939f-47bf-a84f-6839009d8694 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:57.143321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b178fe-939f-47bf-a84f-6839009d8694 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:57.144759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 551/1085] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:57.150802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c1e1a-44d0-4975-9db2-22be999dcce9 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:57.151365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c1e1a-44d0-4975-9db2-22be999dcce9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:57.151365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0c1e1a-44d0-4975-9db2-22be999dcce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:57.169952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4e0c1e1a-44d0-4975-9db2-22be999dcce9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:57.171136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4e0c1e1a-44d0-4975-9db2-22be999dcce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:57.236523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 535/1086] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:57:57.506025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b178fe-939f-47bf-a84f-6839009d8694 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:57.606938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b178fe-939f-47bf-a84f-6839009d8694 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:57.639667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b37c2d7-4bd1-4366-9c99-ff6adca16e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:57.640314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b37c2d7-4bd1-4366-9c99-ff6adca16e50 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:57.640597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b37c2d7-4bd1-4366-9c99-ff6adca16e50 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:57.640868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b37c2d7-4bd1-4366-9c99-ff6adca16e50 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:57.651370 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b37c2d7-4bd1-4366-9c99-ff6adca16e50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3b37c2d7-4bd1-4366-9c99-ff6adca16e50 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:57.656130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:57.657073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:57.658616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 552/1087] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 15 13:57:57 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:57.663826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0c1e1a-44d0-4975-9db2-22be999dcce9 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:57.701791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:57.761080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0c1e1a-44d0-4975-9db2-22be999dcce9 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:57.794013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:57.794625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:57.794882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:57.795149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:57.795835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:57.796136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:57.797474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:57.805805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 536/1088] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 15 13:57:57 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:57.811583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b37c2d7-4bd1-4366-9c99-ff6adca16e50 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:57.837601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:57.911161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b37c2d7-4bd1-4366-9c99-ff6adca16e50 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:57.944640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:57.945268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:57.945548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:57.945824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:57.957045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:57.958557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:57.973373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:57.974542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:57.975441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:57.975441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:57.975441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:57.975441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 553/1089] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:57.980248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:58.075857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:58.080505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8ae760-7e27-4a67-8bcd-88c8347a2ac4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:58.113847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:58.114444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:58.114722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:58.114992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:58.115625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:58.115954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:58.117309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:58.129770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 537/1090] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:58.135368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:58.211355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:58.233265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c3fd42-c7af-418f-afb6-513984e1163a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:58.266436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4619751c-18d9-4d99-8a16-0cd67947cb6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:58.267912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619751c-18d9-4d99-8a16-0cd67947cb6e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:58.268202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619751c-18d9-4d99-8a16-0cd67947cb6e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:58.268473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619751c-18d9-4d99-8a16-0cd67947cb6e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:58.276635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4619751c-18d9-4d99-8a16-0cd67947cb6e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:58.278055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4619751c-18d9-4d99-8a16-0cd67947cb6e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-4619751c-18d9-4d99-8a16-0cd67947cb6e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:57:58.289253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:58.290284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:58.291262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 554/1091] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 15 13:57:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:58.296268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:58.398818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd97d717-618d-45a9-a34f-b9554df67c9c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:58.425111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b85770-8b1d-468f-92a7-44d309a3484c None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:58.425493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b85770-8b1d-468f-92a7-44d309a3484c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:58.425831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b85770-8b1d-468f-92a7-44d309a3484c None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:58.427465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 555/1092] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:57:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:57:58.432240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:58.432395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3b06c5-397b-40e4-932c-891cf8898129 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:58.432663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3b06c5-397b-40e4-932c-891cf8898129 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:58.432836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:58.432940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3b06c5-397b-40e4-932c-891cf8898129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:58.433111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:58.433379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:58.434098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:58.434400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:58.435816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:58.451521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5e3b06c5-397b-40e4-932c-891cf8898129 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:57:58.452718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5e3b06c5-397b-40e4-932c-891cf8898129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:57:58.454111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 538/1093] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 15 13:57:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:57:58.460323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:58.478506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:58.520027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 556/1094] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:57:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:57:58.559474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6af77-0551-4a4d-9ca0-e0cb798f238d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:58.593603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5127d8c7-501c-4e21-a9bc-915e19f71433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:58.594198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127d8c7-501c-4e21-a9bc-915e19f71433 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:58.594479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127d8c7-501c-4e21-a9bc-915e19f71433 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:58.594753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127d8c7-501c-4e21-a9bc-915e19f71433 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:58.660046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5127d8c7-501c-4e21-a9bc-915e19f71433 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:58.661393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5127d8c7-501c-4e21-a9bc-915e19f71433 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:58.678813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:58.756352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 539/1095] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 15 13:57:58.762105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3b06c5-397b-40e4-932c-891cf8898129 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:58.788228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5127d8c7-501c-4e21-a9bc-915e19f71433 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:58.826999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:58.862424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3b06c5-397b-40e4-932c-891cf8898129 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:58.890631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5127d8c7-501c-4e21-a9bc-915e19f71433 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:58.897611 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:58.898246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:58.898525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:58.898800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:58.908880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:58.909150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:58.910423 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:58.926562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae43585a-001a-4619-98a5-9d01a49114fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:57:58.926854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:58.928237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:57:58.929292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:57:58.929292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:57:58.929292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:58.929466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae43585a-001a-4619-98a5-9d01a49114fa None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:58.929466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae43585a-001a-4619-98a5-9d01a49114fa None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:58.929466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae43585a-001a-4619-98a5-9d01a49114fa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:58.929659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 557/1096] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:57:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:58.934709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:58.940234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae43585a-001a-4619-98a5-9d01a49114fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-ae43585a-001a-4619-98a5-9d01a49114fa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:57:58.945027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:58.945966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:57:58.946808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 540/1097] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:57:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:58.952074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae43585a-001a-4619-98a5-9d01a49114fa None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:59.038258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f417fe-6957-4895-97fc-2654593b18e3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.053997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae43585a-001a-4619-98a5-9d01a49114fa None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.054741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:59.072150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.072762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.073043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.073318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.073964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:59.074269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.075640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.087526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 558/1098] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:57:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:59.088608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.089210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.089489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.089762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.090423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:59.090727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.092117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.093168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:59.105707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 541/1099] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:57:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:59.106744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:59.148352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:59.192045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b1edf-f217-45ba-8f52-96275c79cbc4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.207586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec2b60a-a65f-46de-b966-c91ef4a19bbb None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.226315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.226941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.227220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.227491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.234043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.236324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.241396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ff0242-fc60-442c-9fbe-ceca92809175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.242024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff0242-fc60-442c-9fbe-ceca92809175 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.242304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff0242-fc60-442c-9fbe-ceca92809175 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.242577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ff0242-fc60-442c-9fbe-ceca92809175 None admin] PATH_INFO: `/v3/users/2e00af28daae4a8d8e3eaf0fd7508f76` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.243724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ff0242-fc60-442c-9fbe-ceca92809175 None admin] RBAC: Authorizing `identity:get_user(user_id=2e00af28daae4a8d8e3eaf0fd7508f76)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.245133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ff0242-fc60-442c-9fbe-ceca92809175 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.246700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 542/1100] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:57:59 2022] GET /identity/v3/users/2e00af28daae4a8d8e3eaf0fd7508f76 => generated 285 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:57:59.248450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:57:59.249361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:57:59.250268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 559/1101] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:57:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:57:59.253084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ff0242-fc60-442c-9fbe-ceca92809175 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:59.255954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:59.295558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:59.356828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ff0242-fc60-442c-9fbe-ceca92809175 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.359904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8419b09-a8c6-4b2d-82fd-d819f1a7dea4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.365498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:59.390024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ba6f28-3b7a-4d81-a123-42e4f0eb27de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.390624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ba6f28-3b7a-4d81-a123-42e4f0eb27de None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.390900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ba6f28-3b7a-4d81-a123-42e4f0eb27de None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.391171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ba6f28-3b7a-4d81-a123-42e4f0eb27de None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.394504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.395839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.395839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.395839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.396376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:59.396683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.398116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.410582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 560/1102] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:57:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:59.416648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:59.459439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ba6f28-3b7a-4d81-a123-42e4f0eb27de None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.460947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ba6f28-3b7a-4d81-a123-42e4f0eb27de None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.512319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:59.516560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fd5d2-f440-4f00-a5d1-897eedff3e51 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.550563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.551159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.551435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.551721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.552691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] RBAC: Adding query filter params (role.id=d511c5452170401cba1f1162075b62be, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=ce6744790db74f0aaffc6eb269bf6737) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:59.552995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.554451 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.558738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 543/1103] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:57:59 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 15 13:57:59.560074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 561/1104] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:59 2022] GET /identity/v3/role_assignments?user.id=ce6744790db74f0aaffc6eb269bf6737&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=d511c5452170401cba1f1162075b62be => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:59.564739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ba6f28-3b7a-4d81-a123-42e4f0eb27de None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:59.575397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:57:59.669704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ba6f28-3b7a-4d81-a123-42e4f0eb27de None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.707247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546f1d4-516d-4629-b6a7-f448a11cc15f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.707924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546f1d4-516d-4629-b6a7-f448a11cc15f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.708206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546f1d4-516d-4629-b6a7-f448a11cc15f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.708482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546f1d4-516d-4629-b6a7-f448a11cc15f None admin] PATH_INFO: `/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.709697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546f1d4-516d-4629-b6a7-f448a11cc15f None admin] RBAC: Authorizing `identity:get_project(project_id=cc6f33ffe30f4ef0941a62f2a03e9810)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.711092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546f1d4-516d-4629-b6a7-f448a11cc15f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.712835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 544/1105] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:57:59 2022] GET /identity/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:57:59.719049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:57:59.824236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3dbb8e-cda5-4954-ad56-3bc18beae624 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:57:59.859433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a7d02-a2e5-44ec-a652-f2d19110ec0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:57:59.860081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a7d02-a2e5-44ec-a652-f2d19110ec0a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:57:59.860363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a7d02-a2e5-44ec-a652-f2d19110ec0a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:57:59.860838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a7d02-a2e5-44ec-a652-f2d19110ec0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:57:59.861582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9a7d02-a2e5-44ec-a652-f2d19110ec0a None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=cc6f33ffe30f4ef0941a62f2a03e9810, user.id=2e00af28daae4a8d8e3eaf0fd7508f76) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:57:59.862690 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9a7d02-a2e5-44ec-a652-f2d19110ec0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:57:59.863357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9a7d02-a2e5-44ec-a652-f2d19110ec0a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:57:59.869333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 562/1106] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:57:59 2022] GET /identity/v3/role_assignments?user.id=2e00af28daae4a8d8e3eaf0fd7508f76&scope.project.id=cc6f33ffe30f4ef0941a62f2a03e9810&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:02.069125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a6497-6778-44ea-88a0-3bdfc34bc497 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:02.069655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a6497-6778-44ea-88a0-3bdfc34bc497 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:02.069655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a6497-6778-44ea-88a0-3bdfc34bc497 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:02.070997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 545/1107] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:02.076261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c3784b-9b5a-4aac-ad2f-cf83f1ffefbe None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:02.076697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c3784b-9b5a-4aac-ad2f-cf83f1ffefbe None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:02.076817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c3784b-9b5a-4aac-ad2f-cf83f1ffefbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:02.095298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-20c3784b-9b5a-4aac-ad2f-cf83f1ffefbe None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:02.096519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-20c3784b-9b5a-4aac-ad2f-cf83f1ffefbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:02.116960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:02.163236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 563/1108] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:02.436006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0a6497-6778-44ea-88a0-3bdfc34bc497 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:02.512063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:02.535365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0a6497-6778-44ea-88a0-3bdfc34bc497 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:02.569273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ac8b5c-ef4c-4898-92e6-55f285c5037c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:02.569878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac8b5c-ef4c-4898-92e6-55f285c5037c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:02.570150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac8b5c-ef4c-4898-92e6-55f285c5037c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:02.570415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac8b5c-ef4c-4898-92e6-55f285c5037c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:02.571492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ac8b5c-ef4c-4898-92e6-55f285c5037c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:02.572941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ac8b5c-ef4c-4898-92e6-55f285c5037c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:02.574456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 546/1109] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:58:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:02.580319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c3784b-9b5a-4aac-ad2f-cf83f1ffefbe None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:02.676454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c3784b-9b5a-4aac-ad2f-cf83f1ffefbe None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:02.708753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a43ff46-0b0c-438b-99fa-43b8c04e6de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:02.709351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43ff46-0b0c-438b-99fa-43b8c04e6de6 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:02.709630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43ff46-0b0c-438b-99fa-43b8c04e6de6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:02.709900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43ff46-0b0c-438b-99fa-43b8c04e6de6 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:02.710556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a43ff46-0b0c-438b-99fa-43b8c04e6de6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:02.712010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a43ff46-0b0c-438b-99fa-43b8c04e6de6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:02.732574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 564/1110] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:58:02 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:58:04.903024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dade50-f275-4608-a343-b07a44a56f56 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:04.903582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dade50-f275-4608-a343-b07a44a56f56 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:04.903582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dade50-f275-4608-a343-b07a44a56f56 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:04.904946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 547/1111] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:04.910664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf1065-07f1-4a87-a423-92fe745521f6 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:04.911093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf1065-07f1-4a87-a423-92fe745521f6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:04.911219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf1065-07f1-4a87-a423-92fe745521f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:04.930298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3fcf1065-07f1-4a87-a423-92fe745521f6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:04.931489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3fcf1065-07f1-4a87-a423-92fe745521f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:04.939414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:04.998396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 565/1112] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:05.267857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dade50-f275-4608-a343-b07a44a56f56 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:05.366645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dade50-f275-4608-a343-b07a44a56f56 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:05.399427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0765c67-b96a-4234-9151-886ee88cc69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:05.400072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0765c67-b96a-4234-9151-886ee88cc69f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:05.400354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0765c67-b96a-4234-9151-886ee88cc69f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:05.400630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0765c67-b96a-4234-9151-886ee88cc69f None admin] PATH_INFO: `/v3/roles/a0256b6f3d6c4517be88b324fcbc1145` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:05.402012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0765c67-b96a-4234-9151-886ee88cc69f None admin] RBAC: Authorizing `identity:get_role(role_id=a0256b6f3d6c4517be88b324fcbc1145)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:05.403362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0765c67-b96a-4234-9151-886ee88cc69f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:05.404709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 548/1113] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:05 2022] GET /identity/v3/roles/a0256b6f3d6c4517be88b324fcbc1145 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:05.411188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcf1065-07f1-4a87-a423-92fe745521f6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:05.505143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcf1065-07f1-4a87-a423-92fe745521f6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:05.537658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95990e71-4de6-44ba-a374-fb6c8b25266c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:05.538209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95990e71-4de6-44ba-a374-fb6c8b25266c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:05.538485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95990e71-4de6-44ba-a374-fb6c8b25266c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:05.538755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95990e71-4de6-44ba-a374-fb6c8b25266c None admin] PATH_INFO: `/v3/users/7fbff56c81364b2793876ae0e8d08d48` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:05.549727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:05.551962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95990e71-4de6-44ba-a374-fb6c8b25266c None admin] RBAC: Authorizing `identity:get_user(user_id=7fbff56c81364b2793876ae0e8d08d48)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:05.553740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95990e71-4de6-44ba-a374-fb6c8b25266c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:05.555653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 566/1114] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:05 2022] GET /identity/v3/users/7fbff56c81364b2793876ae0e8d08d48 => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:05.561250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0765c67-b96a-4234-9151-886ee88cc69f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:05.576604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:05.658954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0765c67-b96a-4234-9151-886ee88cc69f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:05.691779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832f7373-1e5f-49d2-bd82-2dcccdbaf828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:05.692375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832f7373-1e5f-49d2-bd82-2dcccdbaf828 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:05.692657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832f7373-1e5f-49d2-bd82-2dcccdbaf828 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:05.692934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832f7373-1e5f-49d2-bd82-2dcccdbaf828 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:05.757100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832f7373-1e5f-49d2-bd82-2dcccdbaf828 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:05.758524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832f7373-1e5f-49d2-bd82-2dcccdbaf828 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:05.771895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:05.854227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 549/1115] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:58:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:58:05.859982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95990e71-4de6-44ba-a374-fb6c8b25266c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:05.961408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95990e71-4de6-44ba-a374-fb6c8b25266c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:05.995031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ac81d1-bf2d-4113-8b11-7d2eab5abe37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:05.995649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac81d1-bf2d-4113-8b11-7d2eab5abe37 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:05.995959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac81d1-bf2d-4113-8b11-7d2eab5abe37 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:05.996236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac81d1-bf2d-4113-8b11-7d2eab5abe37 None admin] PATH_INFO: `/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:05.997343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ac81d1-bf2d-4113-8b11-7d2eab5abe37 None admin] RBAC: Authorizing `identity:get_project(project_id=cc6f33ffe30f4ef0941a62f2a03e9810)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:05.998778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ac81d1-bf2d-4113-8b11-7d2eab5abe37 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:06.000430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 567/1116] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:05 2022] GET /identity/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:06.006877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832f7373-1e5f-49d2-bd82-2dcccdbaf828 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:06.103041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832f7373-1e5f-49d2-bd82-2dcccdbaf828 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:06.115625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:06.135902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1747e0b2-a45e-45c4-a7d7-865ed0598289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:06.136493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1747e0b2-a45e-45c4-a7d7-865ed0598289 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:06.136772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1747e0b2-a45e-45c4-a7d7-865ed0598289 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:06.137046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1747e0b2-a45e-45c4-a7d7-865ed0598289 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:06.137772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1747e0b2-a45e-45c4-a7d7-865ed0598289 None admin] RBAC: Adding query filter params (role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=cc6f33ffe30f4ef0941a62f2a03e9810, user.id=7fbff56c81364b2793876ae0e8d08d48) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:06.138082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1747e0b2-a45e-45c4-a7d7-865ed0598289 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:06.139486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1747e0b2-a45e-45c4-a7d7-865ed0598289 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:06.145310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 550/1117] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:58:06 2022] GET /identity/v3/role_assignments?user.id=7fbff56c81364b2793876ae0e8d08d48&scope.project.id=cc6f33ffe30f4ef0941a62f2a03e9810&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:08.258089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814788d1-c5a1-41b1-917c-8ad7142c52e6 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:08.258618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814788d1-c5a1-41b1-917c-8ad7142c52e6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:08.258618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814788d1-c5a1-41b1-917c-8ad7142c52e6 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:08.260012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 568/1118] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:08.265171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7557c87-d471-4dc1-959f-4388d19714ae None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:08.265588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7557c87-d471-4dc1-959f-4388d19714ae None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:08.265720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7557c87-d471-4dc1-959f-4388d19714ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:08.284962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e7557c87-d471-4dc1-959f-4388d19714ae None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:08.286132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e7557c87-d471-4dc1-959f-4388d19714ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:08.351593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 551/1119] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:08.622951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814788d1-c5a1-41b1-917c-8ad7142c52e6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:08.689658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:08.723974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814788d1-c5a1-41b1-917c-8ad7142c52e6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:08.757183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173ad036-bca6-466c-a42f-470cc576a4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:08.757804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ad036-bca6-466c-a42f-470cc576a4cd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:08.758086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ad036-bca6-466c-a42f-470cc576a4cd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:08.758362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ad036-bca6-466c-a42f-470cc576a4cd None admin] PATH_INFO: `/v3/roles/a0256b6f3d6c4517be88b324fcbc1145` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:08.759767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173ad036-bca6-466c-a42f-470cc576a4cd None admin] RBAC: Authorizing `identity:get_role(role_id=a0256b6f3d6c4517be88b324fcbc1145)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:08.761134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173ad036-bca6-466c-a42f-470cc576a4cd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:08.762430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 569/1120] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:08 2022] GET /identity/v3/roles/a0256b6f3d6c4517be88b324fcbc1145 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:08.769085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7557c87-d471-4dc1-959f-4388d19714ae None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:08.864265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7557c87-d471-4dc1-959f-4388d19714ae None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:08.871291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:08.909007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc42213b-8ce2-4701-81d4-2db0b1e49faa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:08.909569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc42213b-8ce2-4701-81d4-2db0b1e49faa None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:08.909848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc42213b-8ce2-4701-81d4-2db0b1e49faa None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:08.910121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc42213b-8ce2-4701-81d4-2db0b1e49faa None admin] PATH_INFO: `/v3/users/7fbff56c81364b2793876ae0e8d08d48` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:08.911402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc42213b-8ce2-4701-81d4-2db0b1e49faa None admin] RBAC: Authorizing `identity:get_user(user_id=7fbff56c81364b2793876ae0e8d08d48)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:08.912854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc42213b-8ce2-4701-81d4-2db0b1e49faa None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:08.914736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 552/1121] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:08 2022] GET /identity/v3/users/7fbff56c81364b2793876ae0e8d08d48 => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:08.920540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173ad036-bca6-466c-a42f-470cc576a4cd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:09.018904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173ad036-bca6-466c-a42f-470cc576a4cd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:09.053181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab8cb1f-d9aa-4e2c-9c87-9bdc9aeeae2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:09.053806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8cb1f-d9aa-4e2c-9c87-9bdc9aeeae2d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:09.054090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8cb1f-d9aa-4e2c-9c87-9bdc9aeeae2d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:09.054366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8cb1f-d9aa-4e2c-9c87-9bdc9aeeae2d None admin] PATH_INFO: `/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:09.055480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab8cb1f-d9aa-4e2c-9c87-9bdc9aeeae2d None admin] RBAC: Authorizing `identity:get_project(project_id=cc6f33ffe30f4ef0941a62f2a03e9810)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:09.056951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab8cb1f-d9aa-4e2c-9c87-9bdc9aeeae2d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:09.058559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 570/1122] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:08 2022] GET /identity/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:09.065368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc42213b-8ce2-4701-81d4-2db0b1e49faa None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:09.160531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc42213b-8ce2-4701-81d4-2db0b1e49faa None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:09.193108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8101b3db-5ea5-4b9d-9486-485cd3a24394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:09.193738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101b3db-5ea5-4b9d-9486-485cd3a24394 None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:09.194018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101b3db-5ea5-4b9d-9486-485cd3a24394 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:09.194292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101b3db-5ea5-4b9d-9486-485cd3a24394 None admin] PATH_INFO: `/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810/users/7fbff56c81364b2793876ae0e8d08d48/roles/a0256b6f3d6c4517be88b324fcbc1145` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:09.196366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8101b3db-5ea5-4b9d-9486-485cd3a24394 None admin] RBAC: Authorizing `identity:create_grant(project_id=cc6f33ffe30f4ef0941a62f2a03e9810, user_id=7fbff56c81364b2793876ae0e8d08d48, role_id=a0256b6f3d6c4517be88b324fcbc1145)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:09.197771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8101b3db-5ea5-4b9d-9486-485cd3a24394 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:09.209953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 553/1123] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:58:09 2022] PUT /identity/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810/users/7fbff56c81364b2793876ae0e8d08d48/roles/a0256b6f3d6c4517be88b324fcbc1145 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:58:11.321968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a486b24-3ad8-47d1-830d-82206101a1bb None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:11.322395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a486b24-3ad8-47d1-830d-82206101a1bb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:11.322515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a486b24-3ad8-47d1-830d-82206101a1bb None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:11.323889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 571/1124] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:11.329931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f463c34-f033-4ef1-9e26-c20f8ecb61e7 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:11.330499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f463c34-f033-4ef1-9e26-c20f8ecb61e7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:11.330499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f463c34-f033-4ef1-9e26-c20f8ecb61e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:11.348716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-1f463c34-f033-4ef1-9e26-c20f8ecb61e7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:11.349900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1f463c34-f033-4ef1-9e26-c20f8ecb61e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:11.375076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:11.415375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 554/1125] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:11.685183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a486b24-3ad8-47d1-830d-82206101a1bb None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:11.689691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:11.784766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a486b24-3ad8-47d1-830d-82206101a1bb None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:11.817475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8156ff-99cd-4af1-b303-d6c73c8e9102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:11.818086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8156ff-99cd-4af1-b303-d6c73c8e9102 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:11.818364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8156ff-99cd-4af1-b303-d6c73c8e9102 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:11.818636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8156ff-99cd-4af1-b303-d6c73c8e9102 None admin] PATH_INFO: `/v3/roles/a0256b6f3d6c4517be88b324fcbc1145` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:11.820017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8156ff-99cd-4af1-b303-d6c73c8e9102 None admin] RBAC: Authorizing `identity:get_role(role_id=a0256b6f3d6c4517be88b324fcbc1145)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:11.821368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8156ff-99cd-4af1-b303-d6c73c8e9102 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:11.822649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 572/1126] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:11 2022] GET /identity/v3/roles/a0256b6f3d6c4517be88b324fcbc1145 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:11.828385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f463c34-f033-4ef1-9e26-c20f8ecb61e7 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:11.924151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f463c34-f033-4ef1-9e26-c20f8ecb61e7 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:11.957514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0adf05-8a5a-4e40-a3e2-6eb477ee1a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:11.958127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0adf05-8a5a-4e40-a3e2-6eb477ee1a34 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:11.958406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0adf05-8a5a-4e40-a3e2-6eb477ee1a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:11.958679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0adf05-8a5a-4e40-a3e2-6eb477ee1a34 None admin] PATH_INFO: `/v3/users/7fbff56c81364b2793876ae0e8d08d48` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:11.959798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0adf05-8a5a-4e40-a3e2-6eb477ee1a34 None admin] RBAC: Authorizing `identity:get_user(user_id=7fbff56c81364b2793876ae0e8d08d48)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:11.961208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0adf05-8a5a-4e40-a3e2-6eb477ee1a34 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:11.962749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 555/1127] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:11 2022] GET /identity/v3/users/7fbff56c81364b2793876ae0e8d08d48 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:11.968548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8156ff-99cd-4af1-b303-d6c73c8e9102 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:12.038899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:12.068984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8156ff-99cd-4af1-b303-d6c73c8e9102 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:12.101683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879e0b4-51e7-45c7-a936-9fdc52ef069e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:12.102277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879e0b4-51e7-45c7-a936-9fdc52ef069e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:12.102558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879e0b4-51e7-45c7-a936-9fdc52ef069e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:12.102833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879e0b4-51e7-45c7-a936-9fdc52ef069e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:12.168312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7879e0b4-51e7-45c7-a936-9fdc52ef069e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:12.169749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7879e0b4-51e7-45c7-a936-9fdc52ef069e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:12.236936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:12.266294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 573/1128] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:58:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:58:12.272246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0adf05-8a5a-4e40-a3e2-6eb477ee1a34 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:12.302899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:12.369280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0adf05-8a5a-4e40-a3e2-6eb477ee1a34 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:12.401671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53cf7dd-4f7d-443e-b783-3ace944c703f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:12.402289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53cf7dd-4f7d-443e-b783-3ace944c703f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:12.402563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53cf7dd-4f7d-443e-b783-3ace944c703f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:12.402832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53cf7dd-4f7d-443e-b783-3ace944c703f None admin] PATH_INFO: `/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:12.403944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53cf7dd-4f7d-443e-b783-3ace944c703f None admin] RBAC: Authorizing `identity:get_project(project_id=cc6f33ffe30f4ef0941a62f2a03e9810)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:12.405354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53cf7dd-4f7d-443e-b783-3ace944c703f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:12.406956 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 556/1129] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:12 2022] GET /identity/v3/projects/cc6f33ffe30f4ef0941a62f2a03e9810 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:12.413672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879e0b4-51e7-45c7-a936-9fdc52ef069e None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:12.509997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879e0b4-51e7-45c7-a936-9fdc52ef069e None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:12.542648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc864375-f3d5-4aaa-8776-381d514a79e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:12.543239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc864375-f3d5-4aaa-8776-381d514a79e7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:12.543517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc864375-f3d5-4aaa-8776-381d514a79e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:12.543818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc864375-f3d5-4aaa-8776-381d514a79e7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:12.544536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc864375-f3d5-4aaa-8776-381d514a79e7 None admin] RBAC: Adding query filter params (role.id=a0256b6f3d6c4517be88b324fcbc1145, scope.project.id=cc6f33ffe30f4ef0941a62f2a03e9810, user.id=7fbff56c81364b2793876ae0e8d08d48) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:12.544843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc864375-f3d5-4aaa-8776-381d514a79e7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:12.546256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc864375-f3d5-4aaa-8776-381d514a79e7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:12.551922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 574/1130] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:58:12 2022] GET /identity/v3/role_assignments?user.id=7fbff56c81364b2793876ae0e8d08d48&scope.project.id=cc6f33ffe30f4ef0941a62f2a03e9810&role.id=a0256b6f3d6c4517be88b324fcbc1145 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:14.719431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36347762-f84f-4ea5-93f1-949f174ad3f0 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:14.720630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36347762-f84f-4ea5-93f1-949f174ad3f0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:14.720774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36347762-f84f-4ea5-93f1-949f174ad3f0 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:14.722165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 557/1131] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:14.727354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2173e9-ca12-4122-b561-f776f2113ed3 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:14.727743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2173e9-ca12-4122-b561-f776f2113ed3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:14.727938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2173e9-ca12-4122-b561-f776f2113ed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:14.746357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6b2173e9-ca12-4122-b561-f776f2113ed3 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:14.747538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6b2173e9-ca12-4122-b561-f776f2113ed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:14.775276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:14.813938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 575/1132] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:15.095298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36347762-f84f-4ea5-93f1-949f174ad3f0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:15.189913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:15.193991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36347762-f84f-4ea5-93f1-949f174ad3f0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:15.226585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17397621-8173-45f3-bcb4-931c7e58d546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:15.227191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17397621-8173-45f3-bcb4-931c7e58d546 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:15.227464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17397621-8173-45f3-bcb4-931c7e58d546 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:15.227753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17397621-8173-45f3-bcb4-931c7e58d546 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:15.228838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17397621-8173-45f3-bcb4-931c7e58d546 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:15.230237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17397621-8173-45f3-bcb4-931c7e58d546 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:15.231780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 558/1133] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:58:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:15.238044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2173e9-ca12-4122-b561-f776f2113ed3 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:15.335162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2173e9-ca12-4122-b561-f776f2113ed3 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:15.368238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88befcd-b6f9-4491-8305-d1284686c390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:15.368788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88befcd-b6f9-4491-8305-d1284686c390 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:15.369070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88befcd-b6f9-4491-8305-d1284686c390 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:15.369349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88befcd-b6f9-4491-8305-d1284686c390 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:15.370005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88befcd-b6f9-4491-8305-d1284686c390 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:15.371435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88befcd-b6f9-4491-8305-d1284686c390 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:15.398331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 576/1134] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:58:15 2022] POST /identity/v3/projects => generated 311 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:58:17.585615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e1f15-0b61-45dc-a8ce-a91fe303db86 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:17.586117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e1f15-0b61-45dc-a8ce-a91fe303db86 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:17.586117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e1f15-0b61-45dc-a8ce-a91fe303db86 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:17.587385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 559/1135] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:17.592591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf356a6-a22d-4a19-85af-1c28b4d71058 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:17.592985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf356a6-a22d-4a19-85af-1c28b4d71058 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:17.593146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf356a6-a22d-4a19-85af-1c28b4d71058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:17.611484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-eaf356a6-a22d-4a19-85af-1c28b4d71058 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:17.612693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-eaf356a6-a22d-4a19-85af-1c28b4d71058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:17.625418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:17.679914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 577/1136] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:58:17.949892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e1f15-0b61-45dc-a8ce-a91fe303db86 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:18.046898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e1f15-0b61-45dc-a8ce-a91fe303db86 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:18.079874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de78d89f-014e-4b7b-bb8d-d4b48d2db860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:18.080474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de78d89f-014e-4b7b-bb8d-d4b48d2db860 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:18.080750 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de78d89f-014e-4b7b-bb8d-d4b48d2db860 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:18.081016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de78d89f-014e-4b7b-bb8d-d4b48d2db860 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:18.082091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de78d89f-014e-4b7b-bb8d-d4b48d2db860 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:18.083493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de78d89f-014e-4b7b-bb8d-d4b48d2db860 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:18.085045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 560/1137] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:58:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:18.090671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf356a6-a22d-4a19-85af-1c28b4d71058 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:18.186385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf356a6-a22d-4a19-85af-1c28b4d71058 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:18.219114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c91c65a-3779-4bd8-9597-b60cd16be68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:18.219843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c91c65a-3779-4bd8-9597-b60cd16be68b None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:18.220191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c91c65a-3779-4bd8-9597-b60cd16be68b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:18.220481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c91c65a-3779-4bd8-9597-b60cd16be68b None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:18.221125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c91c65a-3779-4bd8-9597-b60cd16be68b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:18.222551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c91c65a-3779-4bd8-9597-b60cd16be68b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:18.245080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 578/1138] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:58:18 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:58:20.413342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975cafd-43a0-4b37-a66f-46e92763871c None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:20.413893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975cafd-43a0-4b37-a66f-46e92763871c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:20.413893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975cafd-43a0-4b37-a66f-46e92763871c None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:20.415196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 561/1139] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 13:58:20.420431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8f9463-ede2-46fa-8823-cc2ab51fd885 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:20.420827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8f9463-ede2-46fa-8823-cc2ab51fd885 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:20.420968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8f9463-ede2-46fa-8823-cc2ab51fd885 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:20.429399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:20.441398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-af8f9463-ede2-46fa-8823-cc2ab51fd885 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:20.442573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-af8f9463-ede2-46fa-8823-cc2ab51fd885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:20.510317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 579/1140] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:20.792413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c975cafd-43a0-4b37-a66f-46e92763871c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:20.827801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:20.892215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c975cafd-43a0-4b37-a66f-46e92763871c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:20.923756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89f9474-ae10-45cd-ab47-80c39d683a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:20.924374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89f9474-ae10-45cd-ab47-80c39d683a38 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:20.924660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89f9474-ae10-45cd-ab47-80c39d683a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:20.924920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89f9474-ae10-45cd-ab47-80c39d683a38 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:20.935064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89f9474-ae10-45cd-ab47-80c39d683a38 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-f89f9474-ae10-45cd-ab47-80c39d683a38 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:58:20.940598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:58:20.941706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:58:20.942545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 562/1141] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:58:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:58:20.947854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8f9463-ede2-46fa-8823-cc2ab51fd885 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:21.045456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8f9463-ede2-46fa-8823-cc2ab51fd885 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:21.078524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:21.079731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:21.080194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:21.080469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:21.081126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:21.081424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:21.082794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:21.084113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:21.091386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 580/1142] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:58:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:21.097407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89f9474-ae10-45cd-ab47-80c39d683a38 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:21.189589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:21.195672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89f9474-ae10-45cd-ab47-80c39d683a38 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:21.228364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc32c5-b28d-46cb-9cf3-c43bf335c137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:21.228979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc32c5-b28d-46cb-9cf3-c43bf335c137 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:21.229261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc32c5-b28d-46cb-9cf3-c43bf335c137 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:21.229536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc32c5-b28d-46cb-9cf3-c43bf335c137 None admin] PATH_INFO: `/v3/users/f150a1ec54d44ba4bf08d9ef84b35289` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:21.242699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fc32c5-b28d-46cb-9cf3-c43bf335c137 None admin] RBAC: Authorizing `identity:get_user(user_id=f150a1ec54d44ba4bf08d9ef84b35289)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:21.244167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fc32c5-b28d-46cb-9cf3-c43bf335c137 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:21.245745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 563/1143] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:21 2022] GET /identity/v3/users/f150a1ec54d44ba4bf08d9ef84b35289 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:21.252442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:21.349512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99299ea4-55b7-4374-acf7-8e8df18e8ce4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:21.382762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba5ceb3-7f0d-4597-8fd1-eb8e98aaa255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:21.383365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba5ceb3-7f0d-4597-8fd1-eb8e98aaa255 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:21.383646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba5ceb3-7f0d-4597-8fd1-eb8e98aaa255 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:21.383948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba5ceb3-7f0d-4597-8fd1-eb8e98aaa255 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:21.443399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:21.450133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba5ceb3-7f0d-4597-8fd1-eb8e98aaa255 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:21.451575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba5ceb3-7f0d-4597-8fd1-eb8e98aaa255 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:21.545790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 581/1144] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:58:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:58:21.552195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc32c5-b28d-46cb-9cf3-c43bf335c137 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:21.648768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc32c5-b28d-46cb-9cf3-c43bf335c137 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:21.681294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c80ee5-67ae-4153-83a7-8c7f1dacbe03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:21.681919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c80ee5-67ae-4153-83a7-8c7f1dacbe03 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:21.682199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c80ee5-67ae-4153-83a7-8c7f1dacbe03 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:21.682475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c80ee5-67ae-4153-83a7-8c7f1dacbe03 None admin] PATH_INFO: `/v3/projects/3346adb52fcf4178881be39422e3e4db` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:21.683583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c80ee5-67ae-4153-83a7-8c7f1dacbe03 None admin] RBAC: Authorizing `identity:get_project(project_id=3346adb52fcf4178881be39422e3e4db)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:21.685057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c80ee5-67ae-4153-83a7-8c7f1dacbe03 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:21.686663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 564/1145] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:21 2022] GET /identity/v3/projects/3346adb52fcf4178881be39422e3e4db => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:21.693387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba5ceb3-7f0d-4597-8fd1-eb8e98aaa255 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:21.784825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:21.790945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba5ceb3-7f0d-4597-8fd1-eb8e98aaa255 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:21.823994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78986718-ae4f-44a2-981f-13cd70f556b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:21.824592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78986718-ae4f-44a2-981f-13cd70f556b1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:21.824869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78986718-ae4f-44a2-981f-13cd70f556b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:21.825141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78986718-ae4f-44a2-981f-13cd70f556b1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:21.825857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78986718-ae4f-44a2-981f-13cd70f556b1 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=3346adb52fcf4178881be39422e3e4db, user.id=f150a1ec54d44ba4bf08d9ef84b35289) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:21.826164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78986718-ae4f-44a2-981f-13cd70f556b1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:21.827586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78986718-ae4f-44a2-981f-13cd70f556b1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:21.833363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 582/1146] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:58:21 2022] GET /identity/v3/role_assignments?user.id=f150a1ec54d44ba4bf08d9ef84b35289&scope.project.id=3346adb52fcf4178881be39422e3e4db&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:23.944874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e422c3ea-1249-4878-932b-c9c8e4d6d9de None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:23.945401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e422c3ea-1249-4878-932b-c9c8e4d6d9de None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:23.945401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e422c3ea-1249-4878-932b-c9c8e4d6d9de None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:23.946739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 565/1147] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:23.951767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebd3581-b63c-44b3-b799-49f5db6519ee None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:23.952136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebd3581-b63c-44b3-b799-49f5db6519ee None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:23.952314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebd3581-b63c-44b3-b799-49f5db6519ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:23.971312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-bebd3581-b63c-44b3-b799-49f5db6519ee None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:23.972502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bebd3581-b63c-44b3-b799-49f5db6519ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:24.037493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 583/1148] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:24.310073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e422c3ea-1249-4878-932b-c9c8e4d6d9de None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:24.334602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:24.409333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e422c3ea-1249-4878-932b-c9c8e4d6d9de None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:24.442638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612b0c5f-e110-4c31-bc20-772749e8bf88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:24.443247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b0c5f-e110-4c31-bc20-772749e8bf88 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:24.443526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b0c5f-e110-4c31-bc20-772749e8bf88 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:24.443825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b0c5f-e110-4c31-bc20-772749e8bf88 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:24.453512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612b0c5f-e110-4c31-bc20-772749e8bf88 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-612b0c5f-e110-4c31-bc20-772749e8bf88 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:58:24.458445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:58:24.459500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:58:24.460584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 566/1149] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:58:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:58:24.465539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebd3581-b63c-44b3-b799-49f5db6519ee None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:24.552818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:24.562628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebd3581-b63c-44b3-b799-49f5db6519ee None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:24.595408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:24.596073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:24.596355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:24.596629 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:24.597290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:24.597590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:24.598945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:24.606694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 584/1150] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:58:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:24.613217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612b0c5f-e110-4c31-bc20-772749e8bf88 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:24.693925 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:24.711809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612b0c5f-e110-4c31-bc20-772749e8bf88 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:24.745439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0664015c-3494-4b92-91f5-33a735023c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:24.746071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0664015c-3494-4b92-91f5-33a735023c13 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:24.746338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0664015c-3494-4b92-91f5-33a735023c13 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:24.746612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0664015c-3494-4b92-91f5-33a735023c13 None admin] PATH_INFO: `/v3/users/f150a1ec54d44ba4bf08d9ef84b35289` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:24.747742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0664015c-3494-4b92-91f5-33a735023c13 None admin] RBAC: Authorizing `identity:get_user(user_id=f150a1ec54d44ba4bf08d9ef84b35289)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:24.749184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0664015c-3494-4b92-91f5-33a735023c13 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:24.750764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 567/1151] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:24 2022] GET /identity/v3/users/f150a1ec54d44ba4bf08d9ef84b35289 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:24.756733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:24.856561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5111293-db86-4a08-a7ea-43b28425f829 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:24.888599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8420402-c62d-466c-b70c-5627176feb33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:24.889218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8420402-c62d-466c-b70c-5627176feb33 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:24.889506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8420402-c62d-466c-b70c-5627176feb33 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:24.889779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8420402-c62d-466c-b70c-5627176feb33 None admin] PATH_INFO: `/v3/projects/3346adb52fcf4178881be39422e3e4db` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:24.890892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8420402-c62d-466c-b70c-5627176feb33 None admin] RBAC: Authorizing `identity:get_project(project_id=3346adb52fcf4178881be39422e3e4db)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:24.892986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8420402-c62d-466c-b70c-5627176feb33 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:24.894594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 585/1152] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:24 2022] GET /identity/v3/projects/3346adb52fcf4178881be39422e3e4db => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:24.900576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0664015c-3494-4b92-91f5-33a735023c13 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:24.996404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0664015c-3494-4b92-91f5-33a735023c13 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:25.029460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c56dc4-6579-4231-a0b4-7d706922ef71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:25.030107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c56dc4-6579-4231-a0b4-7d706922ef71 None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:25.030388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c56dc4-6579-4231-a0b4-7d706922ef71 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:25.030663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c56dc4-6579-4231-a0b4-7d706922ef71 None admin] PATH_INFO: `/v3/projects/3346adb52fcf4178881be39422e3e4db/users/f150a1ec54d44ba4bf08d9ef84b35289/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:25.033154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c56dc4-6579-4231-a0b4-7d706922ef71 None admin] RBAC: Authorizing `identity:create_grant(project_id=3346adb52fcf4178881be39422e3e4db, user_id=f150a1ec54d44ba4bf08d9ef84b35289, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:25.034572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c56dc4-6579-4231-a0b4-7d706922ef71 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:25.047609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 568/1153] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:58:24 2022] PUT /identity/v3/projects/3346adb52fcf4178881be39422e3e4db/users/f150a1ec54d44ba4bf08d9ef84b35289/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:58:27.160930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd5aa0-d5a7-47de-b013-730d53bf8a1b None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:27.161499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd5aa0-d5a7-47de-b013-730d53bf8a1b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:27.161499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fd5aa0-d5a7-47de-b013-730d53bf8a1b None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:27.162809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 586/1154] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:27.168122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7901ff-a0f8-4a96-a981-6ad9fa586bbd None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:27.168502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7901ff-a0f8-4a96-a981-6ad9fa586bbd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:27.168701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7901ff-a0f8-4a96-a981-6ad9fa586bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:27.186688 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-3b7901ff-a0f8-4a96-a981-6ad9fa586bbd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:27.187895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3b7901ff-a0f8-4a96-a981-6ad9fa586bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:27.190696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:27.253914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 569/1155] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:27.522785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fd5aa0-d5a7-47de-b013-730d53bf8a1b None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:27.545215 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:27.623270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fd5aa0-d5a7-47de-b013-730d53bf8a1b None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:27.655485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d326ad7-5bf4-4ee0-8481-159256f79922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:27.656125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d326ad7-5bf4-4ee0-8481-159256f79922 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:27.656407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d326ad7-5bf4-4ee0-8481-159256f79922 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:27.656681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d326ad7-5bf4-4ee0-8481-159256f79922 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:27.667580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d326ad7-5bf4-4ee0-8481-159256f79922 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-0d326ad7-5bf4-4ee0-8481-159256f79922 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:58:27.672287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:58:27.674028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:58:27.675026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 587/1156] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:58:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:58:27.680355 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7901ff-a0f8-4a96-a981-6ad9fa586bbd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:27.774822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7901ff-a0f8-4a96-a981-6ad9fa586bbd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:27.807024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:27.807605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:27.807906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:27.808175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:27.809081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:27.809381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:27.810728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:27.818867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 570/1157] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:58:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:27.824896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d326ad7-5bf4-4ee0-8481-159256f79922 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:27.905136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:27.925435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d326ad7-5bf4-4ee0-8481-159256f79922 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:27.958882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c45002-667a-45b2-bc5b-cf9a6c9ffd23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:27.959496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c45002-667a-45b2-bc5b-cf9a6c9ffd23 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:27.959796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c45002-667a-45b2-bc5b-cf9a6c9ffd23 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:27.960072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c45002-667a-45b2-bc5b-cf9a6c9ffd23 None admin] PATH_INFO: `/v3/users/f150a1ec54d44ba4bf08d9ef84b35289` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:27.961170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c45002-667a-45b2-bc5b-cf9a6c9ffd23 None admin] RBAC: Authorizing `identity:get_user(user_id=f150a1ec54d44ba4bf08d9ef84b35289)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:27.962601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c45002-667a-45b2-bc5b-cf9a6c9ffd23 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:27.964231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 588/1158] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:27 2022] GET /identity/v3/users/f150a1ec54d44ba4bf08d9ef84b35289 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:27.970575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:27.973393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:28.068874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c512bb-60ac-4c3c-bd23-80d9824a8686 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:28.101381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4727ac-3c86-4a21-8957-a211801dbfb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:28.101977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4727ac-3c86-4a21-8957-a211801dbfb4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:28.102258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4727ac-3c86-4a21-8957-a211801dbfb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:28.102532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4727ac-3c86-4a21-8957-a211801dbfb4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:28.167296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4727ac-3c86-4a21-8957-a211801dbfb4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:28.168755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4727ac-3c86-4a21-8957-a211801dbfb4 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:28.170494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:28.264626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 571/1159] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:58:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:58:28.271402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c45002-667a-45b2-bc5b-cf9a6c9ffd23 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:28.369142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c45002-667a-45b2-bc5b-cf9a6c9ffd23 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:28.401818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a030f2-7a23-4b0f-b6e5-25556520f3cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:28.402435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a030f2-7a23-4b0f-b6e5-25556520f3cd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:28.402716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a030f2-7a23-4b0f-b6e5-25556520f3cd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:28.402990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a030f2-7a23-4b0f-b6e5-25556520f3cd None admin] PATH_INFO: `/v3/projects/3346adb52fcf4178881be39422e3e4db` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:28.404122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a030f2-7a23-4b0f-b6e5-25556520f3cd None admin] RBAC: Authorizing `identity:get_project(project_id=3346adb52fcf4178881be39422e3e4db)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:28.405551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a030f2-7a23-4b0f-b6e5-25556520f3cd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:28.407307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 589/1160] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:28 2022] GET /identity/v3/projects/3346adb52fcf4178881be39422e3e4db => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:28.413431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4727ac-3c86-4a21-8957-a211801dbfb4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:28.508928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4727ac-3c86-4a21-8957-a211801dbfb4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:28.510407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:28.541945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df7630a-00ae-4b75-b796-520e1cb81385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:28.542536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7630a-00ae-4b75-b796-520e1cb81385 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:28.542814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7630a-00ae-4b75-b796-520e1cb81385 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:28.543084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7630a-00ae-4b75-b796-520e1cb81385 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:28.543824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df7630a-00ae-4b75-b796-520e1cb81385 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=3346adb52fcf4178881be39422e3e4db, user.id=f150a1ec54d44ba4bf08d9ef84b35289) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:28.544134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df7630a-00ae-4b75-b796-520e1cb81385 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:28.545536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df7630a-00ae-4b75-b796-520e1cb81385 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:28.551543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 572/1161] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:58:28 2022] GET /identity/v3/role_assignments?user.id=f150a1ec54d44ba4bf08d9ef84b35289&scope.project.id=3346adb52fcf4178881be39422e3e4db&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:30.706403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50cb99-ebd2-478f-af78-71f48b64f634 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:30.706906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50cb99-ebd2-478f-af78-71f48b64f634 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:30.707013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50cb99-ebd2-478f-af78-71f48b64f634 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:30.708351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 590/1162] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:30.713605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e8321f-8df6-4486-ad9f-f5932584f5c0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:30.714014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e8321f-8df6-4486-ad9f-f5932584f5c0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:30.714153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e8321f-8df6-4486-ad9f-f5932584f5c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:30.732375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e2e8321f-8df6-4486-ad9f-f5932584f5c0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:30.733551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e2e8321f-8df6-4486-ad9f-f5932584f5c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:30.799218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 573/1163] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:31.069443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db50cb99-ebd2-478f-af78-71f48b64f634 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:31.087742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:31.170022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db50cb99-ebd2-478f-af78-71f48b64f634 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:31.203130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29404105-3e2e-490c-ba14-1247c5ce9fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:31.203740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29404105-3e2e-490c-ba14-1247c5ce9fdd None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:31.204227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29404105-3e2e-490c-ba14-1247c5ce9fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:31.204692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29404105-3e2e-490c-ba14-1247c5ce9fdd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:31.269230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29404105-3e2e-490c-ba14-1247c5ce9fdd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:31.270662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29404105-3e2e-490c-ba14-1247c5ce9fdd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:31.287341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:31.368989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 591/1164] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:58:31 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:58:31.374777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e8321f-8df6-4486-ad9f-f5932584f5c0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:31.465167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:31.471917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e8321f-8df6-4486-ad9f-f5932584f5c0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:31.504319 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a848e766-c1dd-4e52-ad19-99ef8a6a4469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:31.504928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848e766-c1dd-4e52-ad19-99ef8a6a4469 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:31.505208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848e766-c1dd-4e52-ad19-99ef8a6a4469 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:31.505478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848e766-c1dd-4e52-ad19-99ef8a6a4469 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:31.511875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a848e766-c1dd-4e52-ad19-99ef8a6a4469 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:31.513293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a848e766-c1dd-4e52-ad19-99ef8a6a4469 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-a848e766-c1dd-4e52-ad19-99ef8a6a4469 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:58:31.524959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:58:31.526136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:58:31.527374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 574/1165] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 13:58:31 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:58:31.532634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29404105-3e2e-490c-ba14-1247c5ce9fdd None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:31.629395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29404105-3e2e-490c-ba14-1247c5ce9fdd None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:31.645780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:31.663769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b46c4f-58b3-4097-85af-6f9479767002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:31.664372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b46c4f-58b3-4097-85af-6f9479767002 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:31.664656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b46c4f-58b3-4097-85af-6f9479767002 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:31.664937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b46c4f-58b3-4097-85af-6f9479767002 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:31.665579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b46c4f-58b3-4097-85af-6f9479767002 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:31.665886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b46c4f-58b3-4097-85af-6f9479767002 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:31.667243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b46c4f-58b3-4097-85af-6f9479767002 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:31.674205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 592/1166] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 15 13:58:31 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:33.791646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79294ecc-22a0-4c70-9a6e-d9326a03589b None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:33.792228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79294ecc-22a0-4c70-9a6e-d9326a03589b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:33.792228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79294ecc-22a0-4c70-9a6e-d9326a03589b None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:33.793545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 575/1167] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:33.799511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec19587-20e1-4093-a918-7e0c09181766 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:33.799913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec19587-20e1-4093-a918-7e0c09181766 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:33.800096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec19587-20e1-4093-a918-7e0c09181766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:33.818388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5ec19587-20e1-4093-a918-7e0c09181766 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:33.819573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5ec19587-20e1-4093-a918-7e0c09181766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:33.884449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 593/1168] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:34.152166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79294ecc-22a0-4c70-9a6e-d9326a03589b None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:34.249056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79294ecc-22a0-4c70-9a6e-d9326a03589b None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:34.282818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d678fb7-4f69-4f33-b95d-e0348023e975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:34.283399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d678fb7-4f69-4f33-b95d-e0348023e975 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:34.283676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d678fb7-4f69-4f33-b95d-e0348023e975 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:34.283975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d678fb7-4f69-4f33-b95d-e0348023e975 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:34.284642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d678fb7-4f69-4f33-b95d-e0348023e975 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:34.285988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d678fb7-4f69-4f33-b95d-e0348023e975 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:34.301339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 576/1169] 10.222.0.27 () {60 vars in 1138 bytes} [Thu Sep 15 13:58:34 2022] POST /identity/v3/domains => generated 267 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:58:36.461958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa346a0-50e1-4ec9-b8f1-e84d2cb41e1e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:36.462485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa346a0-50e1-4ec9-b8f1-e84d2cb41e1e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:36.462485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa346a0-50e1-4ec9-b8f1-e84d2cb41e1e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:36.463885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 594/1170] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:36.469105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb6096-7641-4878-927d-9aada6ffdc29 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:36.469550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb6096-7641-4878-927d-9aada6ffdc29 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:36.469650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb6096-7641-4878-927d-9aada6ffdc29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:36.487334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f3bb6096-7641-4878-927d-9aada6ffdc29 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:36.488527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f3bb6096-7641-4878-927d-9aada6ffdc29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:36.498674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:36.553137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 577/1171] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:36.832122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa346a0-50e1-4ec9-b8f1-e84d2cb41e1e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:36.960379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa346a0-50e1-4ec9-b8f1-e84d2cb41e1e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:36.972242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:37.002750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a067e1ec-8fa6-4058-a25a-7532712ba3d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:37.003531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067e1ec-8fa6-4058-a25a-7532712ba3d1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:37.004093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067e1ec-8fa6-4058-a25a-7532712ba3d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:37.004582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067e1ec-8fa6-4058-a25a-7532712ba3d1 None admin] PATH_INFO: `/v3/domains/163c258793ef4405b02e42f54cd2b24d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:37.019660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a067e1ec-8fa6-4058-a25a-7532712ba3d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=163c258793ef4405b02e42f54cd2b24d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:37.021863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a067e1ec-8fa6-4058-a25a-7532712ba3d1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:37.023907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 595/1172] 10.222.0.27 () {56 vars in 1183 bytes} [Thu Sep 15 13:58:36 2022] GET /identity/v3/domains/163c258793ef4405b02e42f54cd2b24d => generated 267 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:37.029830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bb6096-7641-4878-927d-9aada6ffdc29 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:37.142450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bb6096-7641-4878-927d-9aada6ffdc29 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:37.182331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1976fc7-b771-44f7-85ba-e7bec518efd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:37.182942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1976fc7-b771-44f7-85ba-e7bec518efd6 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:37.183243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1976fc7-b771-44f7-85ba-e7bec518efd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:37.183517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1976fc7-b771-44f7-85ba-e7bec518efd6 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:37.184207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1976fc7-b771-44f7-85ba-e7bec518efd6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:37.185629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1976fc7-b771-44f7-85ba-e7bec518efd6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:37.208393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:37.221346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 578/1173] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 13:58:37 2022] POST /identity/v3/projects => generated 361 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 13:58:39.427715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a61cf-d6bb-4770-8d1b-b53c80b42f69 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:39.428258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a61cf-d6bb-4770-8d1b-b53c80b42f69 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:39.428258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a61cf-d6bb-4770-8d1b-b53c80b42f69 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:39.429859 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 596/1174] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:39.435058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2711-0fef-413c-bf9a-e95d95d2dfc6 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:39.435493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2711-0fef-413c-bf9a-e95d95d2dfc6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:39.435601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2711-0fef-413c-bf9a-e95d95d2dfc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:39.454927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-a92e2711-0fef-413c-bf9a-e95d95d2dfc6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:39.456141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a92e2711-0fef-413c-bf9a-e95d95d2dfc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:39.519463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 579/1175] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 13:58:39.785826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3a61cf-d6bb-4770-8d1b-b53c80b42f69 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:39.885487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3a61cf-d6bb-4770-8d1b-b53c80b42f69 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:39.917562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2549aa6e-ebc5-4384-8ab7-9cf857e709e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:39.918182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549aa6e-ebc5-4384-8ab7-9cf857e709e0 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:39.918459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549aa6e-ebc5-4384-8ab7-9cf857e709e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:39.918729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549aa6e-ebc5-4384-8ab7-9cf857e709e0 None admin] PATH_INFO: `/v3/domains/163c258793ef4405b02e42f54cd2b24d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:39.919850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2549aa6e-ebc5-4384-8ab7-9cf857e709e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=163c258793ef4405b02e42f54cd2b24d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:39.921273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2549aa6e-ebc5-4384-8ab7-9cf857e709e0 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:39.923365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 597/1176] 10.222.0.27 () {56 vars in 1183 bytes} [Thu Sep 15 13:58:39 2022] GET /identity/v3/domains/163c258793ef4405b02e42f54cd2b24d => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:39.930178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e2711-0fef-413c-bf9a-e95d95d2dfc6 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:40.025458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e2711-0fef-413c-bf9a-e95d95d2dfc6 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:40.051097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:40.059220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6e4a23-1286-40cc-92c8-a7da24924fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:40.059859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e4a23-1286-40cc-92c8-a7da24924fa5 None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:40.060142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e4a23-1286-40cc-92c8-a7da24924fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:40.060420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e4a23-1286-40cc-92c8-a7da24924fa5 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:40.061072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6e4a23-1286-40cc-92c8-a7da24924fa5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:40.062483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6e4a23-1286-40cc-92c8-a7da24924fa5 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:40.083571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 580/1177] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:58:39 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:58:42.256835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e3eb82-8ff6-47f7-8f58-293371825aed None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:42.257403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e3eb82-8ff6-47f7-8f58-293371825aed None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:42.257403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e3eb82-8ff6-47f7-8f58-293371825aed None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:42.258704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 598/1178] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:42.263832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716aa7ff-0532-465b-9967-d1c0801ce79c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:42.264268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716aa7ff-0532-465b-9967-d1c0801ce79c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:42.264373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716aa7ff-0532-465b-9967-d1c0801ce79c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:42.283881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-716aa7ff-0532-465b-9967-d1c0801ce79c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:42.285044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-716aa7ff-0532-465b-9967-d1c0801ce79c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:42.349432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 581/1179] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:42.618113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e3eb82-8ff6-47f7-8f58-293371825aed None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:42.658858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:42.719059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e3eb82-8ff6-47f7-8f58-293371825aed None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:42.751928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a516e03-d7b5-4d8d-be25-cf7b78f30a46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:42.752536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a516e03-d7b5-4d8d-be25-cf7b78f30a46 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:42.752812 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a516e03-d7b5-4d8d-be25-cf7b78f30a46 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:42.753078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a516e03-d7b5-4d8d-be25-cf7b78f30a46 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:42.763364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a516e03-d7b5-4d8d-be25-cf7b78f30a46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-2a516e03-d7b5-4d8d-be25-cf7b78f30a46 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:58:42.768188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:58:42.769305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:58:42.770355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 599/1180] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:58:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:58:42.775346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716aa7ff-0532-465b-9967-d1c0801ce79c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:42.870764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716aa7ff-0532-465b-9967-d1c0801ce79c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:42.885167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:42.905757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:42.906357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:42.906640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:42.907372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:42.907640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:42.907971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:42.909323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:42.917176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 582/1181] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:58:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:42.923635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a516e03-d7b5-4d8d-be25-cf7b78f30a46 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:43.022657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:43.026759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a516e03-d7b5-4d8d-be25-cf7b78f30a46 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:43.059457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4561283-2f4c-4f59-b2b5-5276adca45f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:43.060101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4561283-2f4c-4f59-b2b5-5276adca45f9 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:43.060380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4561283-2f4c-4f59-b2b5-5276adca45f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:43.060650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4561283-2f4c-4f59-b2b5-5276adca45f9 None admin] PATH_INFO: `/v3/users/c1b9194ce52c4a1989a1fb442b178c8d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:43.073851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4561283-2f4c-4f59-b2b5-5276adca45f9 None admin] RBAC: Authorizing `identity:get_user(user_id=c1b9194ce52c4a1989a1fb442b178c8d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:43.075310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4561283-2f4c-4f59-b2b5-5276adca45f9 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:43.076915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 600/1182] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:42 2022] GET /identity/v3/users/c1b9194ce52c4a1989a1fb442b178c8d => generated 311 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:43.082791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:43.186394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d234aa4-e355-4b8d-bf2b-4be3550d47ab None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:43.217721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e699aace-4a39-4e7b-9889-67c855df122c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:43.218312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699aace-4a39-4e7b-9889-67c855df122c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:43.218588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699aace-4a39-4e7b-9889-67c855df122c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:43.218858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699aace-4a39-4e7b-9889-67c855df122c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:43.262096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:43.284403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e699aace-4a39-4e7b-9889-67c855df122c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:43.285824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e699aace-4a39-4e7b-9889-67c855df122c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:43.380799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 583/1183] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:58:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:58:43.386725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4561283-2f4c-4f59-b2b5-5276adca45f9 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:43.486485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4561283-2f4c-4f59-b2b5-5276adca45f9 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:43.519234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22513ca-ca6f-450d-b794-2f4563a44ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:43.519885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22513ca-ca6f-450d-b794-2f4563a44ecb None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:43.520166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22513ca-ca6f-450d-b794-2f4563a44ecb None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:43.520440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22513ca-ca6f-450d-b794-2f4563a44ecb None admin] PATH_INFO: `/v3/projects/d5931bd5e6c6443792e07e3849ed07a1` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:43.521543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22513ca-ca6f-450d-b794-2f4563a44ecb None admin] RBAC: Authorizing `identity:get_project(project_id=d5931bd5e6c6443792e07e3849ed07a1)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:43.522965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22513ca-ca6f-450d-b794-2f4563a44ecb None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:43.524589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 601/1184] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:43 2022] GET /identity/v3/projects/d5931bd5e6c6443792e07e3849ed07a1 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:43.530708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e699aace-4a39-4e7b-9889-67c855df122c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:43.603761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:43.626323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e699aace-4a39-4e7b-9889-67c855df122c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:43.659097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844461c4-d2a2-4cf3-b047-ac6140f52e2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:43.659706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844461c4-d2a2-4cf3-b047-ac6140f52e2a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:43.659992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844461c4-d2a2-4cf3-b047-ac6140f52e2a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:43.660265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844461c4-d2a2-4cf3-b047-ac6140f52e2a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:43.660978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844461c4-d2a2-4cf3-b047-ac6140f52e2a None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=d5931bd5e6c6443792e07e3849ed07a1, user.id=c1b9194ce52c4a1989a1fb442b178c8d) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:43.661284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844461c4-d2a2-4cf3-b047-ac6140f52e2a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:43.662683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844461c4-d2a2-4cf3-b047-ac6140f52e2a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:43.667818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 584/1185] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:58:43 2022] GET /identity/v3/role_assignments?user.id=c1b9194ce52c4a1989a1fb442b178c8d&scope.project.id=d5931bd5e6c6443792e07e3849ed07a1&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:45.775735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d33b9-d086-4c45-baa5-459ebeb0c420 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:45.776253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d33b9-d086-4c45-baa5-459ebeb0c420 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:45.776339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d33b9-d086-4c45-baa5-459ebeb0c420 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:45.777621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 602/1186] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:45.783181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9ca52-b6d1-4bee-aa6f-a148a0d90c4b None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:45.783615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9ca52-b6d1-4bee-aa6f-a148a0d90c4b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:45.783792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c9ca52-b6d1-4bee-aa6f-a148a0d90c4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:45.801844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-09c9ca52-b6d1-4bee-aa6f-a148a0d90c4b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:45.803005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-09c9ca52-b6d1-4bee-aa6f-a148a0d90c4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:45.866606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 585/1187] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:46.133357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08d33b9-d086-4c45-baa5-459ebeb0c420 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:46.158593 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:46.234332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08d33b9-d086-4c45-baa5-459ebeb0c420 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:46.267769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ee701e-505d-45db-aa0d-156488e3f076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:46.268380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ee701e-505d-45db-aa0d-156488e3f076 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:46.268654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ee701e-505d-45db-aa0d-156488e3f076 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:46.268920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ee701e-505d-45db-aa0d-156488e3f076 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:46.278528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ee701e-505d-45db-aa0d-156488e3f076 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-98ee701e-505d-45db-aa0d-156488e3f076 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:58:46.284089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:58:46.285161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:58:46.286377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 603/1188] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:58:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:58:46.291229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c9ca52-b6d1-4bee-aa6f-a148a0d90c4b None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:46.358327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:46.388511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c9ca52-b6d1-4bee-aa6f-a148a0d90c4b None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:46.421208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:46.421811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:46.422092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:46.422367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:46.423083 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:46.423386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:46.424860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:46.432856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 586/1189] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:58:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:46.439194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ee701e-505d-45db-aa0d-156488e3f076 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:46.522573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:46.538272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ee701e-505d-45db-aa0d-156488e3f076 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:46.571958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eafd67-3715-4edb-9815-d4b5409c6f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:46.572574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eafd67-3715-4edb-9815-d4b5409c6f80 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:46.572853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eafd67-3715-4edb-9815-d4b5409c6f80 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:46.573127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29eafd67-3715-4edb-9815-d4b5409c6f80 None admin] PATH_INFO: `/v3/users/c1b9194ce52c4a1989a1fb442b178c8d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:46.574229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eafd67-3715-4edb-9815-d4b5409c6f80 None admin] RBAC: Authorizing `identity:get_user(user_id=c1b9194ce52c4a1989a1fb442b178c8d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:46.575656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29eafd67-3715-4edb-9815-d4b5409c6f80 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:46.577241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 604/1190] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:46 2022] GET /identity/v3/users/c1b9194ce52c4a1989a1fb442b178c8d => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:46.583407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:46.680882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2f3562-9140-45b1-82de-56f19935fdd1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:46.709094 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:46.714111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7436b8c7-2646-45ed-bfcd-1859ee4e1e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:46.714735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436b8c7-2646-45ed-bfcd-1859ee4e1e6d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:46.715019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436b8c7-2646-45ed-bfcd-1859ee4e1e6d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:46.715297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436b8c7-2646-45ed-bfcd-1859ee4e1e6d None admin] PATH_INFO: `/v3/projects/d5931bd5e6c6443792e07e3849ed07a1` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:46.716433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7436b8c7-2646-45ed-bfcd-1859ee4e1e6d None admin] RBAC: Authorizing `identity:get_project(project_id=d5931bd5e6c6443792e07e3849ed07a1)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:46.717868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7436b8c7-2646-45ed-bfcd-1859ee4e1e6d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:46.719545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 587/1191] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:46 2022] GET /identity/v3/projects/d5931bd5e6c6443792e07e3849ed07a1 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:46.725466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eafd67-3715-4edb-9815-d4b5409c6f80 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:46.822885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29eafd67-3715-4edb-9815-d4b5409c6f80 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:46.855420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63e1662-118d-488d-b16d-2ab6a59552cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:46.856090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63e1662-118d-488d-b16d-2ab6a59552cd None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:46.856372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63e1662-118d-488d-b16d-2ab6a59552cd None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:46.856646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63e1662-118d-488d-b16d-2ab6a59552cd None admin] PATH_INFO: `/v3/projects/d5931bd5e6c6443792e07e3849ed07a1/users/c1b9194ce52c4a1989a1fb442b178c8d/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:46.858529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63e1662-118d-488d-b16d-2ab6a59552cd None admin] RBAC: Authorizing `identity:create_grant(project_id=d5931bd5e6c6443792e07e3849ed07a1, user_id=c1b9194ce52c4a1989a1fb442b178c8d, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:46.860520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63e1662-118d-488d-b16d-2ab6a59552cd None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:46.872848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 605/1192] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:58:46 2022] PUT /identity/v3/projects/d5931bd5e6c6443792e07e3849ed07a1/users/c1b9194ce52c4a1989a1fb442b178c8d/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:58:48.993487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367e670-0152-47ea-b6e3-e57b500be552 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:48.994016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367e670-0152-47ea-b6e3-e57b500be552 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:48.994016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367e670-0152-47ea-b6e3-e57b500be552 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:48.995370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 588/1193] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:58:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:58:49.001225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04849815-4f7c-4eb5-aa90-9e7f66dc00d1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:49.001675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04849815-4f7c-4eb5-aa90-9e7f66dc00d1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:49.001766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04849815-4f7c-4eb5-aa90-9e7f66dc00d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:49.019736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-04849815-4f7c-4eb5-aa90-9e7f66dc00d1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:58:49.020898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-04849815-4f7c-4eb5-aa90-9e7f66dc00d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:58:49.027938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:49.088855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 606/1194] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:58:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:58:49.362667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1367e670-0152-47ea-b6e3-e57b500be552 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:49.461735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1367e670-0152-47ea-b6e3-e57b500be552 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:49.494575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82769527-5faf-4fb4-be1f-cafe0cc55cd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:49.495183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82769527-5faf-4fb4-be1f-cafe0cc55cd1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:49.495459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82769527-5faf-4fb4-be1f-cafe0cc55cd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:49.495753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82769527-5faf-4fb4-be1f-cafe0cc55cd1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:49.505402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82769527-5faf-4fb4-be1f-cafe0cc55cd1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-82769527-5faf-4fb4-be1f-cafe0cc55cd1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:58:49.510058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:58:49.511151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:58:49.512243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 589/1195] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 13:58:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:58:49.517584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04849815-4f7c-4eb5-aa90-9e7f66dc00d1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:49.614045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04849815-4f7c-4eb5-aa90-9e7f66dc00d1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:49.647172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:49.647798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:49.648079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:49.648350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:49.648999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:49.649299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:49.650657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:49.658485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 607/1196] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 13:58:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:49.664266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82769527-5faf-4fb4-be1f-cafe0cc55cd1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:49.722610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:49.762312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82769527-5faf-4fb4-be1f-cafe0cc55cd1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:49.794247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db829aa-d01e-4686-82d1-62d146c434a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:49.794860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db829aa-d01e-4686-82d1-62d146c434a6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:49.795140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db829aa-d01e-4686-82d1-62d146c434a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:49.795415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db829aa-d01e-4686-82d1-62d146c434a6 None admin] PATH_INFO: `/v3/users/c1b9194ce52c4a1989a1fb442b178c8d` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:49.796553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db829aa-d01e-4686-82d1-62d146c434a6 None admin] RBAC: Authorizing `identity:get_user(user_id=c1b9194ce52c4a1989a1fb442b178c8d)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:49.798689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db829aa-d01e-4686-82d1-62d146c434a6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:49.800288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 590/1197] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 15 13:58:49 2022] GET /identity/v3/users/c1b9194ce52c4a1989a1fb442b178c8d => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:49.806414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:49.810138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:49.904225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c567d7-541a-44c5-a283-30cdbf851da3 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:49.936983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90aa43b9-6862-462c-8671-f77a924fac75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:49.937520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90aa43b9-6862-462c-8671-f77a924fac75 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:49.937798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90aa43b9-6862-462c-8671-f77a924fac75 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:49.938607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90aa43b9-6862-462c-8671-f77a924fac75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:50.003627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90aa43b9-6862-462c-8671-f77a924fac75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:50.005081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90aa43b9-6862-462c-8671-f77a924fac75 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:50.007557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:50.100325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 608/1198] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:58:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 13:58:50.106238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db829aa-d01e-4686-82d1-62d146c434a6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:50.202504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db829aa-d01e-4686-82d1-62d146c434a6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:50.227796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:50.236330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c05637-d101-4615-a5c9-e95e6ece32bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:50.236945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c05637-d101-4615-a5c9-e95e6ece32bb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:50.237223 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c05637-d101-4615-a5c9-e95e6ece32bb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:50.237495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c05637-d101-4615-a5c9-e95e6ece32bb None admin] PATH_INFO: `/v3/projects/d5931bd5e6c6443792e07e3849ed07a1` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:50.238588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c05637-d101-4615-a5c9-e95e6ece32bb None admin] RBAC: Authorizing `identity:get_project(project_id=d5931bd5e6c6443792e07e3849ed07a1)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:50.240040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c05637-d101-4615-a5c9-e95e6ece32bb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:50.241643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 591/1199] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 15 13:58:50 2022] GET /identity/v3/projects/d5931bd5e6c6443792e07e3849ed07a1 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:58:50.247900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90aa43b9-6862-462c-8671-f77a924fac75 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:58:50.344135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90aa43b9-6862-462c-8671-f77a924fac75 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:58:50.347013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:58:50.377544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcfe504-342c-4b21-b6c1-8339379a744e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:58:50.378133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcfe504-342c-4b21-b6c1-8339379a744e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:58:50.378412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcfe504-342c-4b21-b6c1-8339379a744e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:58:50.378683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcfe504-342c-4b21-b6c1-8339379a744e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:58:50.379402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcfe504-342c-4b21-b6c1-8339379a744e None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=d5931bd5e6c6443792e07e3849ed07a1, user.id=c1b9194ce52c4a1989a1fb442b178c8d) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:58:50.379730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcfe504-342c-4b21-b6c1-8339379a744e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:58:50.381151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcfe504-342c-4b21-b6c1-8339379a744e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:58:50.386690 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 609/1200] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:58:50 2022] GET /identity/v3/role_assignments?user.id=c1b9194ce52c4a1989a1fb442b178c8d&scope.project.id=d5931bd5e6c6443792e07e3849ed07a1&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:00.231914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:00.384185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=16 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:46.377189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fe092-d677-4f16-9e1e-b31470c4536b None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:46.377798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fe092-d677-4f16-9e1e-b31470c4536b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:46.377798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fe092-d677-4f16-9e1e-b31470c4536b None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:46.379086 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 592/1201] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:59:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:59:46.384939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df4a7d-af82-457b-a1e4-8a903c7040c1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:46.385469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df4a7d-af82-457b-a1e4-8a903c7040c1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:46.385469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df4a7d-af82-457b-a1e4-8a903c7040c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:46.405482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-b0df4a7d-af82-457b-a1e4-8a903c7040c1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:59:46.407190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b0df4a7d-af82-457b-a1e4-8a903c7040c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:59:46.491850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 610/1202] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:59:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:59:46.762096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380fe092-d677-4f16-9e1e-b31470c4536b None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:46.884055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380fe092-d677-4f16-9e1e-b31470c4536b None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:46.931425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:46.932527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:46.933055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:46.933603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:46.954258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:46.956228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:59:46.974684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:46.976368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:59:46.977631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:59:46.977631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:46.977631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:59:46.977631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 593/1203] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:59:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:59:46.983085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df4a7d-af82-457b-a1e4-8a903c7040c1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:47.116023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df4a7d-af82-457b-a1e4-8a903c7040c1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:47.158430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:47.166291 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204670d6-bc8d-4e84-8fc0-e5b8b1edbde4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:47.167038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204670d6-bc8d-4e84-8fc0-e5b8b1edbde4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:47.167389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204670d6-bc8d-4e84-8fc0-e5b8b1edbde4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:47.167789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204670d6-bc8d-4e84-8fc0-e5b8b1edbde4 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:47.168472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204670d6-bc8d-4e84-8fc0-e5b8b1edbde4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:47.168957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204670d6-bc8d-4e84-8fc0-e5b8b1edbde4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:47.170528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204670d6-bc8d-4e84-8fc0-e5b8b1edbde4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:47.188933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 611/1204] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:59:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:47.190206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:47.257794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:47.334659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda3c364-fe26-4f1e-a0f3-5e7ab47a2bca None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:47.377621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eefcdd9-4192-4388-abdb-53541d80383f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:47.378383 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eefcdd9-4192-4388-abdb-53541d80383f None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:47.378496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eefcdd9-4192-4388-abdb-53541d80383f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:47.378740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eefcdd9-4192-4388-abdb-53541d80383f None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:47.379384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eefcdd9-4192-4388-abdb-53541d80383f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:47.380830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eefcdd9-4192-4388-abdb-53541d80383f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:47.404062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 594/1205] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 15 13:59:47 2022] POST /identity/v3/users => generated 251 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 13:59:49.939421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded3c0f-79e4-4d4b-93b9-13939fcec74e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:49.940140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded3c0f-79e4-4d4b-93b9-13939fcec74e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:49.940140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded3c0f-79e4-4d4b-93b9-13939fcec74e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:49.941922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 612/1206] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:59:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:59:49.948867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c45727-29c4-4bc3-8d58-87de87892dee None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:49.949404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c45727-29c4-4bc3-8d58-87de87892dee None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:49.949595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c45727-29c4-4bc3-8d58-87de87892dee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:49.972805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-a8c45727-29c4-4bc3-8d58-87de87892dee None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:59:49.974126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a8c45727-29c4-4bc3-8d58-87de87892dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:59:50.065014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 595/1207] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:59:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:59:50.370394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded3c0f-79e4-4d4b-93b9-13939fcec74e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:50.493362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded3c0f-79e4-4d4b-93b9-13939fcec74e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:50.532552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702754a-8adb-4ccb-9759-72eb676b2f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:50.533156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702754a-8adb-4ccb-9759-72eb676b2f98 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:50.533437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702754a-8adb-4ccb-9759-72eb676b2f98 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:50.533715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702754a-8adb-4ccb-9759-72eb676b2f98 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:50.547992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d702754a-8adb-4ccb-9759-72eb676b2f98 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d702754a-8adb-4ccb-9759-72eb676b2f98 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:59:50.553577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:50.554559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:59:50.555881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 613/1208] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:59:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:59:50.562084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c45727-29c4-4bc3-8d58-87de87892dee None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:50.604471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:50.668316 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c45727-29c4-4bc3-8d58-87de87892dee None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:50.705597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:50.706218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:50.706507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:50.706770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:50.707440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:50.707775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:50.709148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:50.722974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 596/1209] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:59:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:50.724177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702754a-8adb-4ccb-9759-72eb676b2f98 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:50.782003 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:50.842966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702754a-8adb-4ccb-9759-72eb676b2f98 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:50.879474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:50.880068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:50.880350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:50.880685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:50.893783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:50.895543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:59:50.913399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:50.914431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:59:50.915881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:59:50.915881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:50.915881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:59:50.915881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 614/1210] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:59:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:59:50.920488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:51.033774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:51.038420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfce22fe-9774-4bf3-ac51-fda4eef5e0b9 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:51.083297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:51.083993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:51.084311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:51.084579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:51.085229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:51.085536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:51.086916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:51.099008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 597/1211] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:51.104954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:51.236236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be7888f-5c9f-4c08-8c35-c3d7efb01f1b None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:51.237884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:51.277929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:51.278638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:51.278983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:51.279560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:51.287452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:51.289023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:59:51.302258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:51.303437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 15 13:59:51.304541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:59:51.308857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 615/1212] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:59:51 2022] GET /identity/v3/users/placement => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:59:51.310091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:51.430443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a933bf3-5040-489b-bf1d-62ea36ce2227 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:51.471349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:51.472058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:51.472504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:51.472810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:51.473786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:51.474165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:51.475873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:51.495549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 598/1213] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 15 13:59:51 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 13:59:51.502239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:51.619476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07470976-85b5-49f4-89d2-e630344e0e6c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:51.660883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57223476-66a6-4bc7-88ca-44a275335dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:51.661495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57223476-66a6-4bc7-88ca-44a275335dd5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:51.661773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57223476-66a6-4bc7-88ca-44a275335dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:51.662044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57223476-66a6-4bc7-88ca-44a275335dd5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:51.699420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:51.746553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57223476-66a6-4bc7-88ca-44a275335dd5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:51.749381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57223476-66a6-4bc7-88ca-44a275335dd5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:51.871736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 616/1214] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 13:59:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 15 13:59:51.874742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:51.898767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:52.008581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6b515f-0cc2-412c-9772-5530188e8b04 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:52.051039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:52.051908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:52.051908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:52.052426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:52.065919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:52.067450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:59:52.082409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:52.083496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:59:52.084467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:59:52.084467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:52.084467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:59:52.084467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 599/1215] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:59:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:59:52.089358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57223476-66a6-4bc7-88ca-44a275335dd5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:52.176116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:52.232602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57223476-66a6-4bc7-88ca-44a275335dd5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:52.295735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:52.295735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:52.295735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:52.295735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:52.296197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:52.296197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:52.296197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:52.316533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 617/1216] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:52.317883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:52.419080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:52.463995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612a0ead-1c91-4279-8b57-a6c02eb39f16 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:52.508327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:52.508971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:52.509293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:52.509588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:52.518246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:52.519779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:59:52.531249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:52.532551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 13:59:52.533475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 600/1217] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:59:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:59:52.538393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:52.653916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b928175-6d0a-484c-ac2b-37f9b4c68f5c None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:52.692264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:52.706242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229cbca2-9198-4345-9d51-468ad1e92992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:52.706868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229cbca2-9198-4345-9d51-468ad1e92992 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:52.707141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229cbca2-9198-4345-9d51-468ad1e92992 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:52.707410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229cbca2-9198-4345-9d51-468ad1e92992 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:52.708138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229cbca2-9198-4345-9d51-468ad1e92992 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:52.708437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229cbca2-9198-4345-9d51-468ad1e92992 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:52.709869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229cbca2-9198-4345-9d51-468ad1e92992 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:52.724441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 618/1218] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:59:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:52.731051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:52.838538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c61eafb-8b09-4c97-80ed-214250982d62 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:52.867084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:52.873964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139d10ce-2277-455b-a1cc-d2a321465b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:52.874505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d10ce-2277-455b-a1cc-d2a321465b2e None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:52.874808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d10ce-2277-455b-a1cc-d2a321465b2e None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:52.875061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d10ce-2277-455b-a1cc-d2a321465b2e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:52.876369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139d10ce-2277-455b-a1cc-d2a321465b2e None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=62299b21a55c46d78a62098ee9e3c497) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:52.876681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139d10ce-2277-455b-a1cc-d2a321465b2e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:52.878107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139d10ce-2277-455b-a1cc-d2a321465b2e None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:52.883481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 601/1219] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 13:59:52 2022] GET /identity/v3/role_assignments?user.id=62299b21a55c46d78a62098ee9e3c497&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:55.252146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83b24f5-5d4b-4d46-a6c4-0721b8490d16 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:55.252762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83b24f5-5d4b-4d46-a6c4-0721b8490d16 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:55.252762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83b24f5-5d4b-4d46-a6c4-0721b8490d16 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:55.254039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 619/1220] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:59:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:59:55.260320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afb4c08-3848-412f-9f90-7d2c17b256b8 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:55.260829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afb4c08-3848-412f-9f90-7d2c17b256b8 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:55.260920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afb4c08-3848-412f-9f90-7d2c17b256b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:55.280087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6afb4c08-3848-412f-9f90-7d2c17b256b8 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:59:55.281372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6afb4c08-3848-412f-9f90-7d2c17b256b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:59:55.356487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 602/1221] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:59:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 13:59:55.626884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83b24f5-5d4b-4d46-a6c4-0721b8490d16 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:55.741250 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83b24f5-5d4b-4d46-a6c4-0721b8490d16 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:55.779147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a0038d-9365-4fbc-bb60-f15b9ee53797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:55.779864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a0038d-9365-4fbc-bb60-f15b9ee53797 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:55.780160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a0038d-9365-4fbc-bb60-f15b9ee53797 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:55.780427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a0038d-9365-4fbc-bb60-f15b9ee53797 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:55.790994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a0038d-9365-4fbc-bb60-f15b9ee53797 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a4a0038d-9365-4fbc-bb60-f15b9ee53797 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 13:59:55.795771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:55.796891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:59:55.798461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 620/1222] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 13:59:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:59:55.803562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afb4c08-3848-412f-9f90-7d2c17b256b8 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:55.912656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afb4c08-3848-412f-9f90-7d2c17b256b8 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:55.939041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:55.952712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4043727c-4547-4500-9af2-069083328d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:55.953399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4043727c-4547-4500-9af2-069083328d82 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:55.953846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4043727c-4547-4500-9af2-069083328d82 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:55.954313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4043727c-4547-4500-9af2-069083328d82 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:55.954723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4043727c-4547-4500-9af2-069083328d82 None admin] RBAC: Adding query filter params (name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:55.955039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4043727c-4547-4500-9af2-069083328d82 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:55.956424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4043727c-4547-4500-9af2-069083328d82 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:55.965422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 603/1223] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 13:59:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:55.972555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a0038d-9365-4fbc-bb60-f15b9ee53797 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:56.019484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:56.081746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a0038d-9365-4fbc-bb60-f15b9ee53797 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:56.118666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:56.119294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:56.119575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:56.119889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:56.133130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:56.134626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:59:56.150394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:56.151438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:59:56.158004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:59:56.158004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:56.158004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:59:56.158004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 621/1224] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:59:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 13:59:56.158290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4043727c-4547-4500-9af2-069083328d82 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:56.266630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4043727c-4547-4500-9af2-069083328d82 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:56.303292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:56.303937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:56.304235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:56.304501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:56.305161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:56.305573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:56.307564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:56.324866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 604/1225] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:59:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:56.331934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:56.431643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:56.439507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc1f155-727a-474a-9984-3aed7a871e92 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:56.479160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:56.479821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:56.480102 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:56.480375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:56.488711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:56.490180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 13:59:56.502184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:56.503331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:59:56.504666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 622/1226] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:59:56 2022] GET /identity/v3/users/placement => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:59:56.509101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:56.560736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:56.616755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9191fbd-7a90-4316-8488-2312c2e30148 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:56.651180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:56.651850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:56.652152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:56.652436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:56.653239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:56.653541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:56.654960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:56.674084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 605/1227] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 15 13:59:56 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:56.680131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:56.792971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008751ee-3107-4900-ab33-6cff95a590f2 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:56.831186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:56.831766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:56.832047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:56.832323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:56.843850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:56.846165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 13:59:56.862362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:56.863460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 13:59:56.869076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 13:59:56.869076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 13:59:56.869076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:59:56.869076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 623/1228] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 13:59:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:59:56.870283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:56.980233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b12f5e-41fe-4e03-9ec9-d587852f21c0 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:56.982815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:57.017004 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:57.018255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:57.018255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:57.018255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:57.018786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:57.019085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:57.020482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:57.034290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 606/1229] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 13:59:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:57.040324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:57.050125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:57.148404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa9dca7-a09d-40b0-a5c5-87e881565400 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:57.186613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:57.187242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:57.187517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:57.187867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:57.195492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:57.197103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 13:59:57.209913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 13:59:57.211099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 13:59:57.212202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 624/1230] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 13:59:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 13:59:57.217594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:57.333573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87128100-2aba-4544-af4b-33b84e534ca6 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:57.367604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e684c0-6eee-4c39-a331-2e22a9845f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:57.368243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e684c0-6eee-4c39-a331-2e22a9845f64 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:57.368554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e684c0-6eee-4c39-a331-2e22a9845f64 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:57.368884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e684c0-6eee-4c39-a331-2e22a9845f64 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:57.369684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e684c0-6eee-4c39-a331-2e22a9845f64 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 13:59:57.369992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e684c0-6eee-4c39-a331-2e22a9845f64 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:57.371449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e684c0-6eee-4c39-a331-2e22a9845f64 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:57.385178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 607/1231] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 13:59:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 13:59:57.392424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 13:59:57.464803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 13:59:57.502395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b701b5-0dc1-4ba1-b5eb-a6140bdac9b6 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 13:59:57.536988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb516fab-f364-4b88-88e6-c05269a7466f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 13:59:57.537647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb516fab-f364-4b88-88e6-c05269a7466f None admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:57.537932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb516fab-f364-4b88-88e6-c05269a7466f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:57.538212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb516fab-f364-4b88-88e6-c05269a7466f None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/62299b21a55c46d78a62098ee9e3c497/roles/91265edc38fd488492ee9ae7ab7d2440` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:57.553316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb516fab-f364-4b88-88e6-c05269a7466f None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=62299b21a55c46d78a62098ee9e3c497, role_id=91265edc38fd488492ee9ae7ab7d2440)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 13:59:57.554763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb516fab-f364-4b88-88e6-c05269a7466f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 13:59:57.567744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 625/1232] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 13:59:57 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/62299b21a55c46d78a62098ee9e3c497/roles/91265edc38fd488492ee9ae7ab7d2440 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 13:59:59.795505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67022ca-14f6-44f3-9d7e-fd6b1bab16f9 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:59.796172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67022ca-14f6-44f3-9d7e-fd6b1bab16f9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:59.796172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67022ca-14f6-44f3-9d7e-fd6b1bab16f9 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:59.797790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 608/1233] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 13:59:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 13:59:59.803662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2749c9-151f-4216-9d7c-3d988ee416f2 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 13:59:59.804114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2749c9-151f-4216-9d7c-3d988ee416f2 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 13:59:59.804233 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2749c9-151f-4216-9d7c-3d988ee416f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 13:59:59.826400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7f2749c9-151f-4216-9d7c-3d988ee416f2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 13:59:59.827712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7f2749c9-151f-4216-9d7c-3d988ee416f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 13:59:59.903439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 626/1234] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 13:59:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 14:00:00.191759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67022ca-14f6-44f3-9d7e-fd6b1bab16f9 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:00.221414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:00.312415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67022ca-14f6-44f3-9d7e-fd6b1bab16f9 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:00.351402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b81e48-10bf-457a-86c7-4514bc490831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:00.352065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b81e48-10bf-457a-86c7-4514bc490831 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:00.352345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b81e48-10bf-457a-86c7-4514bc490831 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:00.352625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b81e48-10bf-457a-86c7-4514bc490831 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:00.365076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b81e48-10bf-457a-86c7-4514bc490831 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-49b81e48-10bf-457a-86c7-4514bc490831 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 14:00:00.369946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:00.371046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:00.371950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 609/1235] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 15 14:00:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:00.378655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2749c9-151f-4216-9d7c-3d988ee416f2 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:00.431948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:00.492780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2749c9-151f-4216-9d7c-3d988ee416f2 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:00.529127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:00.529963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:00.530300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:00.530601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:00.532107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:00.532621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:00.534364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:00.544451 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 627/1236] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 15 14:00:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:00.550610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b81e48-10bf-457a-86c7-4514bc490831 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:00.649209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:00.667497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b81e48-10bf-457a-86c7-4514bc490831 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:00.705836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:00.706463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:00.706742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:00.707012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:00.717071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:00.718822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 14:00:00.734430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:00.735614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 14:00:00.736796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 14:00:00.736796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:00.736796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:00.736796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 610/1237] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 14:00:00.741922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:00.858398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf865c9b-c3e1-4e11-ad63-b5815dbf46b5 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:00.870066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:00.900367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:00.901008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:00.901287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:00.901559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:00.902207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:00.902511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:00.904112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:00.917293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 628/1238] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 14:00:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:00.923619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:01.040917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26f0df3-8608-4617-bf09-430f1be0a224 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:01.082897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:01.083557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:01.083872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:01.084189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:01.087293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:01.093174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:01.095612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 14:00:01.107219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:01.108375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:01.109661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 611/1239] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:00 2022] GET /identity/v3/users/placement => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:01.114237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:01.221250 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db09128a-aca3-496f-8362-86418f65a1d4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:01.257661 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:01.258268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:01.258541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:01.258814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:01.259530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:01.259863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:01.261394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:01.280367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 629/1240] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 15 14:00:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:01.286783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:01.400531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83eda637-d82e-4f01-9afc-4baef677a57b None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:01.436188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32ed0b6-8024-498b-af96-0658f9d97a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:01.436788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ed0b6-8024-498b-af96-0658f9d97a35 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:01.437066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ed0b6-8024-498b-af96-0658f9d97a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:01.437328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ed0b6-8024-498b-af96-0658f9d97a35 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:01.505021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:01.511886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32ed0b6-8024-498b-af96-0658f9d97a35 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:01.513358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32ed0b6-8024-498b-af96-0658f9d97a35 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:01.622509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 612/1241] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 14:00:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 15 14:00:01.630126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:01.661934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:01.756122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04017990-9026-4ffd-9ad7-e62bdeb71244 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:01.794225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:01.794867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:01.795145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:01.795433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:01.806966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:01.808564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 14:00:01.824798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:01.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 14:00:01.827337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 14:00:01.827337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:01.827337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:01.827337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 630/1242] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 14:00:01.832365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32ed0b6-8024-498b-af96-0658f9d97a35 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:01.934652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:01.941614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32ed0b6-8024-498b-af96-0658f9d97a35 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:01.987557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:01.988151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:01.988431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:01.988698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:01.989339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:01.989638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:01.990991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:02.003622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 613/1243] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 14:00:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:02.009421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:02.096756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:02.119847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812c7bb6-67af-4a95-8db6-bba0441c12ba None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:02.159797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:02.160427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:02.160706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:02.160999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:02.168408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:02.169863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 14:00:02.182133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:02.183122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:02.184151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 631/1244] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 14:00:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:02.190549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:02.305221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04891a0f-d6b1-4782-8311-c18b949ff7a1 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:02.341727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8015b975-d9dc-43bb-aac8-4c2d10ca378a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:02.342346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8015b975-d9dc-43bb-aac8-4c2d10ca378a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:02.342608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8015b975-d9dc-43bb-aac8-4c2d10ca378a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:02.342877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8015b975-d9dc-43bb-aac8-4c2d10ca378a None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:02.343628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8015b975-d9dc-43bb-aac8-4c2d10ca378a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:02.343908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8015b975-d9dc-43bb-aac8-4c2d10ca378a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:02.345810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8015b975-d9dc-43bb-aac8-4c2d10ca378a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:02.358499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 614/1245] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 14:00:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:02.365291 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:02.481876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cf281-44a4-445f-863f-109cd0e7b8e1 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:02.521020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b560418-5a57-4e96-b408-8bcad32ff42a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:02.521659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b560418-5a57-4e96-b408-8bcad32ff42a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:02.521959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b560418-5a57-4e96-b408-8bcad32ff42a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:02.522274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b560418-5a57-4e96-b408-8bcad32ff42a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:02.523424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b560418-5a57-4e96-b408-8bcad32ff42a None admin] RBAC: Adding query filter params (role.id=91265edc38fd488492ee9ae7ab7d2440, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=62299b21a55c46d78a62098ee9e3c497) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:02.523808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b560418-5a57-4e96-b408-8bcad32ff42a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:02.525623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b560418-5a57-4e96-b408-8bcad32ff42a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:02.532131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 632/1246] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 14:00:02 2022] GET /identity/v3/role_assignments?user.id=62299b21a55c46d78a62098ee9e3c497&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=91265edc38fd488492ee9ae7ab7d2440 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:04.886162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490d4723-e1a4-4cbd-a9bb-592cddf76909 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:04.886748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490d4723-e1a4-4cbd-a9bb-592cddf76909 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:04.886748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490d4723-e1a4-4cbd-a9bb-592cddf76909 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:04.888073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 615/1247] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:00:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:00:04.894543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669b280-ef4b-4f03-8827-880cf4ee07d7 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:04.895099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669b280-ef4b-4f03-8827-880cf4ee07d7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:04.895275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669b280-ef4b-4f03-8827-880cf4ee07d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:04.901694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:04.914913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c669b280-ef4b-4f03-8827-880cf4ee07d7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:00:04.916325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c669b280-ef4b-4f03-8827-880cf4ee07d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:00:05.002599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 633/1248] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:00:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 14:00:05.277694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490d4723-e1a4-4cbd-a9bb-592cddf76909 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:05.296563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:05.388506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490d4723-e1a4-4cbd-a9bb-592cddf76909 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:05.424500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3ba586-b9a0-4e2f-8af1-df512f9487d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:05.425137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3ba586-b9a0-4e2f-8af1-df512f9487d4 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:05.425415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3ba586-b9a0-4e2f-8af1-df512f9487d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:05.425684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3ba586-b9a0-4e2f-8af1-df512f9487d4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:05.437024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3ba586-b9a0-4e2f-8af1-df512f9487d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3c3ba586-b9a0-4e2f-8af1-df512f9487d4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 14:00:05.442061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:05.443059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:05.444130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 616/1249] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 14:00:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:05.449396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c669b280-ef4b-4f03-8827-880cf4ee07d7 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:05.555024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c669b280-ef4b-4f03-8827-880cf4ee07d7 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:05.585291 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:05.590758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:05.591368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:05.591646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:05.591954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:05.592618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:05.592918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:05.594943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:05.603267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 634/1250] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 14:00:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:05.609181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3ba586-b9a0-4e2f-8af1-df512f9487d4 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:05.696285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:05.723566 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3ba586-b9a0-4e2f-8af1-df512f9487d4 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:05.760648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:05.761276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:05.761551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:05.761820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:05.772767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:05.774211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 14:00:05.789070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:05.790059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 14:00:05.791042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 14:00:05.791042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:05.791042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:05.791042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 617/1251] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:05.796756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:05.895706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7609eca-f68a-4204-badf-6b772d856c30 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:05.927915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:05.928518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:05.928800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:05.929072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:05.929713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:05.930015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:05.931624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:05.945122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 635/1252] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 14:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:05.951216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:06.053050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ae9fff-2c68-45b8-bfb1-8b76029b5361 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:06.078496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:06.086517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:06.087140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:06.087414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:06.087695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:06.095132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:06.096574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 14:00:06.107251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:06.108398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:06.110326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 618/1253] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:05 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 15 14:00:06.114929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:06.169768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:06.214561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60630db8-a288-4975-ab89-4b8ca14997d4 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:06.247652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:06.248274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:06.248553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:06.248822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:06.249540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:06.249846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:06.251855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:06.268322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 636/1254] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 15 14:00:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:06.274048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:06.371372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6604ef-1b15-40f9-baca-29310ed95645 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:06.403836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f1e242-5c60-46de-a7af-4c1bb552ed8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:06.404423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f1e242-5c60-46de-a7af-4c1bb552ed8f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:06.404698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f1e242-5c60-46de-a7af-4c1bb552ed8f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:06.404966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f1e242-5c60-46de-a7af-4c1bb552ed8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:06.458036 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:06.470526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f1e242-5c60-46de-a7af-4c1bb552ed8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:06.471981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f1e242-5c60-46de-a7af-4c1bb552ed8f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:06.565908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 619/1255] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 14:00:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 14:00:06.571931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:06.667995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05406318-efe6-477c-9fe7-677205a45314 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:06.681503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:06.701992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:06.702607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:06.702882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:06.703151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:06.713192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:06.714560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 14:00:06.729295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:06.730405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 14:00:06.731329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 14:00:06.731329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:06.731329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:06.731329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 637/1256] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:06.736702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f1e242-5c60-46de-a7af-4c1bb552ed8f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:06.819659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:06.832241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f1e242-5c60-46de-a7af-4c1bb552ed8f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:06.864966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:06.865552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:06.865828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:06.866098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:06.866731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:06.867028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:06.868398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:06.879373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 620/1257] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 14:00:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:06.884832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:06.991843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861cbcaa-61d6-4234-8cac-e5eac035c554 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:07.025046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:07.025671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:07.025953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:07.026229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:07.033452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:07.034890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 14:00:07.045952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:07.047009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:07.048348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 638/1258] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 14:00:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:07.053138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:07.151997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6f77ad-4fa8-4326-b562-5460d2f6fb06 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:07.183929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6384483b-5023-4765-b731-1efdf3272e09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:07.184520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6384483b-5023-4765-b731-1efdf3272e09 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:07.184798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6384483b-5023-4765-b731-1efdf3272e09 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:07.185069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6384483b-5023-4765-b731-1efdf3272e09 None admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:07.185752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6384483b-5023-4765-b731-1efdf3272e09 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:07.186055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6384483b-5023-4765-b731-1efdf3272e09 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:07.187454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6384483b-5023-4765-b731-1efdf3272e09 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:07.199360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 621/1259] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 14:00:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 14:00:07.206132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:07.231192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:07.303917 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9f2c5c-c97b-4b23-831e-0b0026c1b004 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:07.338073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10990e00-057b-410d-b364-9c84ce9de55e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:07.338676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10990e00-057b-410d-b364-9c84ce9de55e None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:07.338960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10990e00-057b-410d-b364-9c84ce9de55e None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:07.339235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10990e00-057b-410d-b364-9c84ce9de55e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:07.339985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10990e00-057b-410d-b364-9c84ce9de55e None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=62299b21a55c46d78a62098ee9e3c497) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:07.340297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10990e00-057b-410d-b364-9c84ce9de55e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:07.341723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10990e00-057b-410d-b364-9c84ce9de55e None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:07.347088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 639/1260] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 14:00:07 2022] GET /identity/v3/role_assignments?user.id=62299b21a55c46d78a62098ee9e3c497&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:09.462724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6906aa-1613-4147-a6b3-82ddf2c46467 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:09.463266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6906aa-1613-4147-a6b3-82ddf2c46467 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:09.463266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6906aa-1613-4147-a6b3-82ddf2c46467 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:09.464638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 622/1261] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:00:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:00:09.470162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cc119d-7e86-4734-aedb-28be757bcfba None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:09.470540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cc119d-7e86-4734-aedb-28be757bcfba None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:09.470704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cc119d-7e86-4734-aedb-28be757bcfba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:09.489029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-48cc119d-7e86-4734-aedb-28be757bcfba None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:00:09.490202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-48cc119d-7e86-4734-aedb-28be757bcfba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:00:09.555559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 640/1262] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:00:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 14:00:09.822926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6906aa-1613-4147-a6b3-82ddf2c46467 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:09.834158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:09.924230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6906aa-1613-4147-a6b3-82ddf2c46467 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:09.959735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e570527-6607-4bfc-ab18-5df328d8f61c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:09.960359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e570527-6607-4bfc-ab18-5df328d8f61c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:09.960638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e570527-6607-4bfc-ab18-5df328d8f61c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:09.960907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e570527-6607-4bfc-ab18-5df328d8f61c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:09.972056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e570527-6607-4bfc-ab18-5df328d8f61c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-9e570527-6607-4bfc-ab18-5df328d8f61c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 14:00:09.976766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:09.977835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:09.978711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 623/1263] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 14:00:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:09.984451 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cc119d-7e86-4734-aedb-28be757bcfba None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:10.005891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:10.087055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cc119d-7e86-4734-aedb-28be757bcfba None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:10.119933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:10.120528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:10.120800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:10.121067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:10.121713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:10.122008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:10.123391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:10.131762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 641/1264] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 14:00:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:10.137592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e570527-6607-4bfc-ab18-5df328d8f61c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:10.201882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:10.234350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e570527-6607-4bfc-ab18-5df328d8f61c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:10.267820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:10.268426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:10.268705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:10.268976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:10.278523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:10.279964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 14:00:10.293898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:10.294962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 14:00:10.296333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 14:00:10.296333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:10.296333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:10.296333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 624/1265] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:10.300971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:10.382440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:10.401466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd95af9-a3fe-41ae-b612-ebbafdf0cef0 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:10.433337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:10.433932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:10.434210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:10.434479 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:10.435114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:10.435415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:10.436804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:10.448640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 642/1266] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 14:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:10.454857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:10.552268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee547f9-772f-4781-b7f8-265b1db1ad15 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:10.571580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:10.585946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:10.586556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:10.586830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:10.587099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:10.594254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:10.595673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 14:00:10.606359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:10.607457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:10.608663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 625/1267] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:10 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:10.613741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:10.710418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a5b179-ec98-43ba-9e19-77c72de41255 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:10.743347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:10.743963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:10.744243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:10.744518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:10.745237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:10.745543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:10.747064 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:10.756238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:10.764940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 643/1268] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 15 14:00:10 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:10.770767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:10.868414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7a580c-3e04-42cd-97cb-cb2fa14c7b2c None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:10.900628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:10.901234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:10.901508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:10.901778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:10.912477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:10.913899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 14:00:10.927524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:10.928489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 14:00:10.929481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 14:00:10.929481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:10.929481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:10.929481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 626/1269] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:10.934712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:11.040579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1079cdfb-c0cd-4db1-9667-f21428a8ab75 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:11.073883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:11.074492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:11.074773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:11.075053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:11.075716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:11.076028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:11.077396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:11.088785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 644/1270] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 14:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:11.094502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:11.124929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:11.193735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ced56d-5abb-4864-a295-42db229b91fb None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:11.225755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:11.226366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:11.226645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:11.226914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:11.233288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:11.235250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 14:00:11.246105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:11.247122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:11.248151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 627/1271] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 14:00:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:11.253619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:11.311260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:11.353818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdda7b3-c7e9-4df6-8880-8219b3560d8f None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:11.387230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e76b06-b1b2-4eae-8d90-48619b9fcf75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:11.387855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e76b06-b1b2-4eae-8d90-48619b9fcf75 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:11.388137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e76b06-b1b2-4eae-8d90-48619b9fcf75 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:11.388410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e76b06-b1b2-4eae-8d90-48619b9fcf75 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:11.389104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e76b06-b1b2-4eae-8d90-48619b9fcf75 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:11.389410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e76b06-b1b2-4eae-8d90-48619b9fcf75 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:11.390870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e76b06-b1b2-4eae-8d90-48619b9fcf75 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:11.402529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 645/1272] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 14:00:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:11.408603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:11.496198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:11.505890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671b7f9c-a792-44e7-8e84-0ddfd8185c1a None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:11.537145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db993515-4083-414d-a1e9-fd36684895a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:11.537778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db993515-4083-414d-a1e9-fd36684895a9 None admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:11.538052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db993515-4083-414d-a1e9-fd36684895a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:11.538320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db993515-4083-414d-a1e9-fd36684895a9 None admin] PATH_INFO: `/v3/projects/64eda23ae35b454786569cfb259d972b/users/62299b21a55c46d78a62098ee9e3c497/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:11.540187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db993515-4083-414d-a1e9-fd36684895a9 None admin] RBAC: Authorizing `identity:create_grant(project_id=64eda23ae35b454786569cfb259d972b, user_id=62299b21a55c46d78a62098ee9e3c497, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:11.541593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db993515-4083-414d-a1e9-fd36684895a9 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:11.553936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 628/1273] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 15 14:00:11 2022] PUT /identity/v3/projects/64eda23ae35b454786569cfb259d972b/users/62299b21a55c46d78a62098ee9e3c497/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:00:13.673200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0c650f-adde-4d81-bf06-b8dcf7b6305a None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:13.673700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0c650f-adde-4d81-bf06-b8dcf7b6305a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:13.673832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0c650f-adde-4d81-bf06-b8dcf7b6305a None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:13.675177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 646/1274] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:00:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:00:13.680332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ef4f9-662a-4ec2-abad-1fcef25746d5 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:13.680808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ef4f9-662a-4ec2-abad-1fcef25746d5 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:13.680919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ef4f9-662a-4ec2-abad-1fcef25746d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:13.699961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-933ef4f9-662a-4ec2-abad-1fcef25746d5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:00:13.701163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-933ef4f9-662a-4ec2-abad-1fcef25746d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:00:13.765926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 629/1275] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:00:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 15 14:00:14.032444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0c650f-adde-4d81-bf06-b8dcf7b6305a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:14.131421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0c650f-adde-4d81-bf06-b8dcf7b6305a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:14.148723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:14.165460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dba0a7-d67b-4bde-84a3-b497ef604ef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:14.166075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dba0a7-d67b-4bde-84a3-b497ef604ef2 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:14.166357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dba0a7-d67b-4bde-84a3-b497ef604ef2 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:14.166628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dba0a7-d67b-4bde-84a3-b497ef604ef2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:14.176669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dba0a7-d67b-4bde-84a3-b497ef604ef2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-04dba0a7-d67b-4bde-84a3-b497ef604ef2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 15 14:00:14.181334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:14.182697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:14.183769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 647/1276] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 15 14:00:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:14.189265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933ef4f9-662a-4ec2-abad-1fcef25746d5 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:14.267491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:14.286172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933ef4f9-662a-4ec2-abad-1fcef25746d5 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:14.319670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:14.320295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:14.320572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:14.320844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:14.321504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:14.321804 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:14.323156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:14.331241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 630/1277] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 15 14:00:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:14.337068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dba0a7-d67b-4bde-84a3-b497ef604ef2 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:14.434014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dba0a7-d67b-4bde-84a3-b497ef604ef2 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:14.466796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:14.467393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:14.467718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:14.467991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:14.477835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:14.479284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 14:00:14.494051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:14.495188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 14:00:14.496441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 14:00:14.496441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:14.496441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:14.496441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 648/1278] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:14.501501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:14.598357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5aaec2-49e9-4fd0-8a7f-4cf59abf5d66 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:14.630620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:14.631216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:14.631500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:14.631802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:14.632442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:14.632743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:14.634107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:14.645889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 631/1279] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 14:00:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:14.651408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:14.681315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:14.748797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f50c98e-e006-4dac-a0ee-10c25b97c146 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:14.781773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:14.782408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:14.782685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:14.782960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:14.790094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:14.791525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 15 14:00:14.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:14.803607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:14.804622 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 649/1280] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:14.809538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:14.814314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:14.906091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204d9e28-3826-47ce-bd0d-23e909a4ab1f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:14.940337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:14.940962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:14.941287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:14.941589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:14.942385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:14.942716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:14.944315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:14.962168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 632/1281] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 15 14:00:14 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:14.968127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:15.057454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:15.067643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7056a09-adcc-4b3b-9a47-4a15e70f0df7 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:15.099869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aaa24c-ad98-4d85-bc74-63497071b6c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:15.100466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aaa24c-ad98-4d85-bc74-63497071b6c9 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:15.100748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aaa24c-ad98-4d85-bc74-63497071b6c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:15.101022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aaa24c-ad98-4d85-bc74-63497071b6c9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:15.165932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aaa24c-ad98-4d85-bc74-63497071b6c9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:15.167307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aaa24c-ad98-4d85-bc74-63497071b6c9 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:15.253258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:15.263628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 650/1282] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 15 14:00:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 15 14:00:15.269833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:15.329923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:15.367338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d738173f-4ed6-4985-b52a-027330fef06f None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:15.400184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:15.400795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:15.401068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:15.401334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:15.411628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:15.413079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 15 14:00:15.427671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:15.428774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 15 14:00:15.429674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 15 14:00:15.429674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 15 14:00:15.429674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:15.429674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 633/1283] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:00:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:15.434685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aaa24c-ad98-4d85-bc74-63497071b6c9 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:15.530614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aaa24c-ad98-4d85-bc74-63497071b6c9 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:15.564104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:15.564697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:15.564971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:15.565236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:15.565867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:15.566164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:15.567511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:15.578238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 651/1284] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 15 14:00:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:15.584558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:15.679747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb998251-1b10-4e73-843f-797f9a7c3370 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:15.693661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:15.714381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:15.714997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:15.715276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:15.715546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:15.722241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:15.723676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 15 14:00:15.735842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: ERROR keystone.server.flask.application Sep 15 14:00:15.738043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 634/1285] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 15 14:00:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:15.742827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:15.792225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:15.842599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f482a53-6969-4060-ae86-204eb08fa33a None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:15.876155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65616170-d0ed-41e7-b4de-f433dcd37e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:15.876760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65616170-d0ed-41e7-b4de-f433dcd37e64 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:15.877054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65616170-d0ed-41e7-b4de-f433dcd37e64 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:15.877318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65616170-d0ed-41e7-b4de-f433dcd37e64 None admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:15.878005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65616170-d0ed-41e7-b4de-f433dcd37e64 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:15.878312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65616170-d0ed-41e7-b4de-f433dcd37e64 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:15.879784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65616170-d0ed-41e7-b4de-f433dcd37e64 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:15.891500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 652/1286] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 15 14:00:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:15.897600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:15.994621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcb6f0f-5612-4545-9409-af99b5a7ac0d None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:16.026889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881020ec-a6f9-48de-b805-1b3125461267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:16.027472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881020ec-a6f9-48de-b805-1b3125461267 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:16.027772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881020ec-a6f9-48de-b805-1b3125461267 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:16.028044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881020ec-a6f9-48de-b805-1b3125461267 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:16.028749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881020ec-a6f9-48de-b805-1b3125461267 None admin] RBAC: Adding query filter params (role.id=ee6d387ccd8c4db1847085e4aa8b762a, scope.project.id=64eda23ae35b454786569cfb259d972b, user.id=62299b21a55c46d78a62098ee9e3c497) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:16.029049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881020ec-a6f9-48de-b805-1b3125461267 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:16.030464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881020ec-a6f9-48de-b805-1b3125461267 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:16.036288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 635/1287] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 15 14:00:15 2022] GET /identity/v3/role_assignments?user.id=62299b21a55c46d78a62098ee9e3c497&scope.project.id=64eda23ae35b454786569cfb259d972b&role.id=ee6d387ccd8c4db1847085e4aa8b762a => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:18.177321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5bc640-5401-43c7-acde-11669b0e2f64 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:18.177864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5bc640-5401-43c7-acde-11669b0e2f64 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:18.177864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5bc640-5401-43c7-acde-11669b0e2f64 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:18.179189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 653/1288] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:00:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:00:18.184713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27941200-e51d-4401-ab88-6915fd025354 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:18.185104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27941200-e51d-4401-ab88-6915fd025354 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:18.185260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27941200-e51d-4401-ab88-6915fd025354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:18.203175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-27941200-e51d-4401-ab88-6915fd025354 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:00:18.204992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-27941200-e51d-4401-ab88-6915fd025354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:00:18.233540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:18.269781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 636/1289] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:00:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 14:00:18.535528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5bc640-5401-43c7-acde-11669b0e2f64 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:18.633948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5bc640-5401-43c7-acde-11669b0e2f64 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:18.644265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:18.667628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:18.668272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:18.668548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:18.668820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:18.669390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:18.670740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:18.680021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:18.681314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:18.682563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 654/1290] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 14:00:18 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:18.687259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27941200-e51d-4401-ab88-6915fd025354 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:18.781508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27941200-e51d-4401-ab88-6915fd025354 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:18.814013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25248be0-2177-4ccb-be7d-4e40bb837dfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:18.814545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25248be0-2177-4ccb-be7d-4e40bb837dfe None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:18.814823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25248be0-2177-4ccb-be7d-4e40bb837dfe None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:18.815093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25248be0-2177-4ccb-be7d-4e40bb837dfe None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:18.815739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25248be0-2177-4ccb-be7d-4e40bb837dfe None admin] RBAC: Adding query filter params (name=placement) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:18.816041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25248be0-2177-4ccb-be7d-4e40bb837dfe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:18.817379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25248be0-2177-4ccb-be7d-4e40bb837dfe None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:18.821984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 637/1291] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Sep 15 14:00:18 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:18.827516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:18.925832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd016a-870d-46dd-9160-f493dbcf9608 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:18.961034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd79aaa-8e6a-45d4-a6cb-f1667f8f33e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:18.961721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd79aaa-8e6a-45d4-a6cb-f1667f8f33e5 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:18.961909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd79aaa-8e6a-45d4-a6cb-f1667f8f33e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:18.962183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd79aaa-8e6a-45d4-a6cb-f1667f8f33e5 None admin] PATH_INFO: `/v3/services` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:18.962812 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd79aaa-8e6a-45d4-a6cb-f1667f8f33e5 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:18.963108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd79aaa-8e6a-45d4-a6cb-f1667f8f33e5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:18.964504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd79aaa-8e6a-45d4-a6cb-f1667f8f33e5 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:18.969772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 655/1292] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Sep 15 14:00:18 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:21.085944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28290bc1-68b5-4b7e-a102-82ab34fc3df0 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:21.086512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28290bc1-68b5-4b7e-a102-82ab34fc3df0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:21.086512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28290bc1-68b5-4b7e-a102-82ab34fc3df0 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:21.087837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 638/1293] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:00:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:00:21.093045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e52bb60-aaf9-4c07-8b50-aef9b207dadb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:21.093479 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e52bb60-aaf9-4c07-8b50-aef9b207dadb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:21.093573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e52bb60-aaf9-4c07-8b50-aef9b207dadb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:21.112273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8e52bb60-aaf9-4c07-8b50-aef9b207dadb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:00:21.113454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8e52bb60-aaf9-4c07-8b50-aef9b207dadb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:00:21.142297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:21.179220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 656/1294] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:00:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 14:00:21.444920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28290bc1-68b5-4b7e-a102-82ab34fc3df0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:21.475007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:21.544371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28290bc1-68b5-4b7e-a102-82ab34fc3df0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:21.577147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d032e25e-7a76-4dd9-b205-0d9a0d5d5e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:21.577738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e25e-7a76-4dd9-b205-0d9a0d5d5e5c None admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:21.578017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e25e-7a76-4dd9-b205-0d9a0d5d5e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:21.578464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e25e-7a76-4dd9-b205-0d9a0d5d5e5c None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:21.579129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d032e25e-7a76-4dd9-b205-0d9a0d5d5e5c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:21.580504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d032e25e-7a76-4dd9-b205-0d9a0d5d5e5c None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:21.594656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 639/1295] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 15 14:00:21 2022] POST /identity/v3/services => generated 246 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:00:23.754057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4be243-2c00-4bc1-b3ce-878ad72bded2 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:23.754575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4be243-2c00-4bc1-b3ce-878ad72bded2 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:23.754692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4be243-2c00-4bc1-b3ce-878ad72bded2 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:23.756009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 657/1296] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:00:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:00:23.761062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a242c57-7fa1-43f8-95cf-4f0064264177 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:23.761454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a242c57-7fa1-43f8-95cf-4f0064264177 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:23.761603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a242c57-7fa1-43f8-95cf-4f0064264177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:23.780238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9a242c57-7fa1-43f8-95cf-4f0064264177 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:00:23.781432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9a242c57-7fa1-43f8-95cf-4f0064264177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:00:23.852299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 640/1297] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:00:23 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 14:00:24.127362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4be243-2c00-4bc1-b3ce-878ad72bded2 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:24.227568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4be243-2c00-4bc1-b3ce-878ad72bded2 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:24.260158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:24.260778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:24.261059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:24.261332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:24.261911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:24.263282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:24.272581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:24.273723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:24.274771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 658/1298] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 14:00:24 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:24.279542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a242c57-7fa1-43f8-95cf-4f0064264177 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:24.297912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:24.377352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a242c57-7fa1-43f8-95cf-4f0064264177 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:24.410432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:24.410968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:24.411247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:24.411519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:24.412175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:24.412478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:24.413818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:24.418686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 641/1299] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Sep 15 14:00:24 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:24.425111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:24.456954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:24.524381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ba7f0a-c96f-492d-b7bc-bae841ffe216 None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:24.556289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2b04eb-5a9d-4d92-9880-b7cb8e6b1a9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:24.557143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b04eb-5a9d-4d92-9880-b7cb8e6b1a9d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:24.557423 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b04eb-5a9d-4d92-9880-b7cb8e6b1a9d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:24.557692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2b04eb-5a9d-4d92-9880-b7cb8e6b1a9d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:24.558383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2b04eb-5a9d-4d92-9880-b7cb8e6b1a9d None admin] RBAC: Adding query filter params (interface=public, service_id=aba33043bf1245ca9a35d81b6ade96a7, region_id=RegionOne) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:24.558687 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2b04eb-5a9d-4d92-9880-b7cb8e6b1a9d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:24.560077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2b04eb-5a9d-4d92-9880-b7cb8e6b1a9d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:24.565779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 659/1300] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 15 14:00:24 2022] GET /identity/v3/endpoints?service_id=aba33043bf1245ca9a35d81b6ade96a7&interface=public®ion_id=RegionOne => generated 194 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:24.571538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:24.649855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:24.667395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11568e6-e177-42df-97a0-51f4c6dceb80 None admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:24.699649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df61fa4-3509-40fd-970b-ac78a4a6fc30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:24.700261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df61fa4-3509-40fd-970b-ac78a4a6fc30 None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:24.700541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df61fa4-3509-40fd-970b-ac78a4a6fc30 None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:24.700815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df61fa4-3509-40fd-970b-ac78a4a6fc30 None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:24.701411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df61fa4-3509-40fd-970b-ac78a4a6fc30 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:24.702755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df61fa4-3509-40fd-970b-ac78a4a6fc30 None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:24.709929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 642/1301] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 15 14:00:24 2022] GET /identity/v3/services => generated 2191 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:00:26.845112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610ee605-6d30-4d00-8959-91e10ee9c4a6 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:26.845645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610ee605-6d30-4d00-8959-91e10ee9c4a6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:26.845734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610ee605-6d30-4d00-8959-91e10ee9c4a6 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:26.847106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 660/1302] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:00:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:00:26.852315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4db508-cebe-47fd-9b3f-d8367db297a7 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:26.852718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4db508-cebe-47fd-9b3f-d8367db297a7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:26.852867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4db508-cebe-47fd-9b3f-d8367db297a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:26.871505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9d4db508-cebe-47fd-9b3f-d8367db297a7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:00:26.872716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9d4db508-cebe-47fd-9b3f-d8367db297a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:00:26.936767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 643/1303] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:00:26 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 15 14:00:27.206434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610ee605-6d30-4d00-8959-91e10ee9c4a6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:27.299422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:27.306472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610ee605-6d30-4d00-8959-91e10ee9c4a6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:27.340229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:27.340842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:27.341119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:27.341386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] PATH_INFO: `/v3/services/placement` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:27.341955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:27.343315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 15 14:00:27.352804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 15 14:00:27.353948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.get_or_create( Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application with Lock( Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self._enter() Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application return self.creator() Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application created_value = creator( Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: ERROR keystone.server.flask.application Sep 15 14:00:27.354948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 661/1304] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 15 14:00:27 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 15 14:00:27.359864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4db508-cebe-47fd-9b3f-d8367db297a7 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:27.432732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:27.455326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4db508-cebe-47fd-9b3f-d8367db297a7 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:27.488272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e333c1-1e5c-4260-83a3-e1aa8d3ac92a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:27.488818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e333c1-1e5c-4260-83a3-e1aa8d3ac92a None admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:27.489101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e333c1-1e5c-4260-83a3-e1aa8d3ac92a None admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:27.489376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e333c1-1e5c-4260-83a3-e1aa8d3ac92a None admin] PATH_INFO: `/v3/services` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:27.490006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e333c1-1e5c-4260-83a3-e1aa8d3ac92a None admin] RBAC: Adding query filter params (name=placement) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:00:27.490311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e333c1-1e5c-4260-83a3-e1aa8d3ac92a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:27.491663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e333c1-1e5c-4260-83a3-e1aa8d3ac92a None admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:27.496727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 644/1305] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Sep 15 14:00:27 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:00:27.503211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:00:27.600153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185bebeb-dbc9-43ad-88e2-3c7a578aaa5d None admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:00:27.632651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25966456-8e37-409f-82b2-b8e87987e4b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:00:27.633237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25966456-8e37-409f-82b2-b8e87987e4b3 None admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:00:27.633513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25966456-8e37-409f-82b2-b8e87987e4b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:00:27.633785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25966456-8e37-409f-82b2-b8e87987e4b3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:00:27.634445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25966456-8e37-409f-82b2-b8e87987e4b3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:00:27.635824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25966456-8e37-409f-82b2-b8e87987e4b3 None admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:00:27.657389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 662/1306] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:00:27 2022] POST /identity/v3/endpoints => generated 331 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:00:37.493745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=30 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:00:37.653004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:31.693782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e6fbba-ed01-45e4-86fc-a1db38e19b54 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:31.694272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e6fbba-ed01-45e4-86fc-a1db38e19b54 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:31.694353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e6fbba-ed01-45e4-86fc-a1db38e19b54 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:31.700033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 645/1307] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:31 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:31.701407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d41284-a9d7-4e2d-8db2-571043e3f024 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:31.701888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d41284-a9d7-4e2d-8db2-571043e3f024 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:31.702059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d41284-a9d7-4e2d-8db2-571043e3f024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:31.793268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-63d41284-a9d7-4e2d-8db2-571043e3f024 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:31.794641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-63d41284-a9d7-4e2d-8db2-571043e3f024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:31.930922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 663/1308] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:32.236139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e504e2-1f67-4557-9b1b-ee75920af5f0 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:32.236481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e504e2-1f67-4557-9b1b-ee75920af5f0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:32.236693 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e504e2-1f67-4557-9b1b-ee75920af5f0 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:32.238442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 646/1309] 10.222.0.27 () {54 vars in 965 bytes} [Thu Sep 15 14:01:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:32.244991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76515ed-70cf-4713-bd93-cc16f87838fe None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:32.245351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76515ed-70cf-4713-bd93-cc16f87838fe None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:32.245704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76515ed-70cf-4713-bd93-cc16f87838fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:32.339699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c76515ed-70cf-4713-bd93-cc16f87838fe None None] MFA Rules not processed for user `922012eacf244f9480c180c2eb9eaf28`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:32.340922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c76515ed-70cf-4713-bd93-cc16f87838fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:32.365087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:32.425933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 664/1310] 10.222.0.27 () {58 vars in 1062 bytes} [Thu Sep 15 14:01:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:32.435220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e504e2-1f67-4557-9b1b-ee75920af5f0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:32.437049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-70e504e2-1f67-4557-9b1b-ee75920af5f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:32.555996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e504e2-1f67-4557-9b1b-ee75920af5f0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:32.639887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc7a263-5fe9-4034-b837-550ab76f5a27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:32.640617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7a263-5fe9-4034-b837-550ab76f5a27 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:32.641062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7a263-5fe9-4034-b837-550ab76f5a27 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:32.641385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7a263-5fe9-4034-b837-550ab76f5a27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:32.643179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-cdc7a263-5fe9-4034-b837-550ab76f5a27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:32.709422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:32.769278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc7a263-5fe9-4034-b837-550ab76f5a27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:32.770751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc7a263-5fe9-4034-b837-550ab76f5a27 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:32.790313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 647/1311] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:01:35.234442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01045d50-fe43-4373-b764-cb4396a5e867 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:35.235571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01045d50-fe43-4373-b764-cb4396a5e867 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:35.235571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01045d50-fe43-4373-b764-cb4396a5e867 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:35.236953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 665/1312] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:35.242095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582aa6c4-bed7-49a5-8de1-5b1bb8ead5a0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:35.242464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582aa6c4-bed7-49a5-8de1-5b1bb8ead5a0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:35.242649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582aa6c4-bed7-49a5-8de1-5b1bb8ead5a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:35.324782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-582aa6c4-bed7-49a5-8de1-5b1bb8ead5a0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:35.325998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-582aa6c4-bed7-49a5-8de1-5b1bb8ead5a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:35.419112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 648/1313] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:01:35.701230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01045d50-fe43-4373-b764-cb4396a5e867 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:35.702757 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-01045d50-fe43-4373-b764-cb4396a5e867 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:35.808093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01045d50-fe43-4373-b764-cb4396a5e867 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:35.881760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:35.884329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c04ba15-27c8-42dc-a1cb-beb5ab6c8d61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:35.884944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04ba15-27c8-42dc-a1cb-beb5ab6c8d61 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:35.885412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04ba15-27c8-42dc-a1cb-beb5ab6c8d61 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:35.885835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04ba15-27c8-42dc-a1cb-beb5ab6c8d61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:35.887727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5c04ba15-27c8-42dc-a1cb-beb5ab6c8d61 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:35.999084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c04ba15-27c8-42dc-a1cb-beb5ab6c8d61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:36.000766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c04ba15-27c8-42dc-a1cb-beb5ab6c8d61 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:36.019400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 666/1314] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:01:38.496870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f96e87d-f3a9-4cd3-a072-03febe974740 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:38.497396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f96e87d-f3a9-4cd3-a072-03febe974740 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:38.497396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f96e87d-f3a9-4cd3-a072-03febe974740 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:38.498807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 649/1315] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:38.504845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475f12d-eb1c-4b30-88b3-fb38b32038e5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:38.505248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475f12d-eb1c-4b30-88b3-fb38b32038e5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:38.505391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475f12d-eb1c-4b30-88b3-fb38b32038e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:38.571051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2475f12d-eb1c-4b30-88b3-fb38b32038e5 None None] MFA Rules not processed for user `054bf47407b34e4a875fbfb8ad1c5a1b`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:38.572183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2475f12d-eb1c-4b30-88b3-fb38b32038e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:38.623794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:38.668298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 667/1316] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:38 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:38.951734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f96e87d-f3a9-4cd3-a072-03febe974740 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:38.953307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1f96e87d-f3a9-4cd3-a072-03febe974740 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:38.965017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:39.059517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f96e87d-f3a9-4cd3-a072-03febe974740 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:39.134010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc994bb-52bf-438d-bfdf-d4c3ca9f6f90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:39.134605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc994bb-52bf-438d-bfdf-d4c3ca9f6f90 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:39.134877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc994bb-52bf-438d-bfdf-d4c3ca9f6f90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:39.135144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc994bb-52bf-438d-bfdf-d4c3ca9f6f90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:39.136760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-efc994bb-52bf-438d-bfdf-d4c3ca9f6f90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:39.198787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:39.245435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc994bb-52bf-438d-bfdf-d4c3ca9f6f90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:39.246850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc994bb-52bf-438d-bfdf-d4c3ca9f6f90 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:39.264791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 650/1317] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:38 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:01:41.968013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06bffd8-bee4-4b3b-bdfe-491c4a28bf0d None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:41.968643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06bffd8-bee4-4b3b-bdfe-491c4a28bf0d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:41.968752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06bffd8-bee4-4b3b-bdfe-491c4a28bf0d None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:41.971236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 668/1318] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:01:41.978029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e64e0d0-3e6b-4db9-a724-3a3e6356a915 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:41.978538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e64e0d0-3e6b-4db9-a724-3a3e6356a915 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:41.978677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e64e0d0-3e6b-4db9-a724-3a3e6356a915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:42.051568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6e64e0d0-3e6b-4db9-a724-3a3e6356a915 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:42.052805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6e64e0d0-3e6b-4db9-a724-3a3e6356a915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:42.158512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 651/1319] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:41 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:42.491323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06bffd8-bee4-4b3b-bdfe-491c4a28bf0d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:42.492902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c06bffd8-bee4-4b3b-bdfe-491c4a28bf0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:42.619730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06bffd8-bee4-4b3b-bdfe-491c4a28bf0d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:42.701163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f394348a-1a2b-454a-a55c-a7860b2f490a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:42.701769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f394348a-1a2b-454a-a55c-a7860b2f490a service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:42.702045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f394348a-1a2b-454a-a55c-a7860b2f490a service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:42.702312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f394348a-1a2b-454a-a55c-a7860b2f490a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:42.704536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f394348a-1a2b-454a-a55c-a7860b2f490a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:42.711793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:42.814685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f394348a-1a2b-454a-a55c-a7860b2f490a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:42.816159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f394348a-1a2b-454a-a55c-a7860b2f490a service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:42.833873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 669/1320] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:01:45.624373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2604d14-134d-4970-b629-8ec8d9e54721 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:45.624878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2604d14-134d-4970-b629-8ec8d9e54721 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:45.624878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2604d14-134d-4970-b629-8ec8d9e54721 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:45.627100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 652/1321] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:45.632227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaf8297-1f6c-421b-9166-37dc08afad84 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:45.632635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaf8297-1f6c-421b-9166-37dc08afad84 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:45.632908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaf8297-1f6c-421b-9166-37dc08afad84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:45.707560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6eaf8297-1f6c-421b-9166-37dc08afad84 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:45.708822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6eaf8297-1f6c-421b-9166-37dc08afad84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:45.770109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:45.813874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 670/1322] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:46.117479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92459e0-69f4-4706-b20f-20f62eba4886 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:46.117763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92459e0-69f4-4706-b20f-20f62eba4886 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:46.118032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92459e0-69f4-4706-b20f-20f62eba4886 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:46.119365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 653/1323] 10.222.0.27 () {54 vars in 965 bytes} [Thu Sep 15 14:01:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:46.125876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc3f90-416b-4195-a4ea-6ac83af64fed None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:46.126166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc3f90-416b-4195-a4ea-6ac83af64fed None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:46.126438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc3f90-416b-4195-a4ea-6ac83af64fed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:46.202690 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-21fc3f90-416b-4195-a4ea-6ac83af64fed None None] MFA Rules not processed for user `922012eacf244f9480c180c2eb9eaf28`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:46.204682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-21fc3f90-416b-4195-a4ea-6ac83af64fed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:46.280058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 671/1324] 10.222.0.27 () {58 vars in 1062 bytes} [Thu Sep 15 14:01:46 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:01:46.289163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92459e0-69f4-4706-b20f-20f62eba4886 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:46.290693 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b92459e0-69f4-4706-b20f-20f62eba4886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:46.317452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:46.403388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92459e0-69f4-4706-b20f-20f62eba4886 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:46.479089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d6a14e-cdca-443e-9764-3b289e05efa9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:46.479717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d6a14e-cdca-443e-9764-3b289e05efa9 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:46.479997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d6a14e-cdca-443e-9764-3b289e05efa9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:46.480271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d6a14e-cdca-443e-9764-3b289e05efa9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:46.481872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-03d6a14e-cdca-443e-9764-3b289e05efa9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:46.552042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:46.590411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d6a14e-cdca-443e-9764-3b289e05efa9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:46.591886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d6a14e-cdca-443e-9764-3b289e05efa9 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:46.610505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 654/1325] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:01:49.055654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143d1938-83f2-41a3-bde4-1e378dec7944 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:49.056134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143d1938-83f2-41a3-bde4-1e378dec7944 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:49.056249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143d1938-83f2-41a3-bde4-1e378dec7944 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:49.057519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 672/1326] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:49.063909 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f99b8b-1a33-431b-a06d-b042946dbf35 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:49.064328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f99b8b-1a33-431b-a06d-b042946dbf35 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:49.064588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f99b8b-1a33-431b-a06d-b042946dbf35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:49.131406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-64f99b8b-1a33-431b-a06d-b042946dbf35 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:49.132626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-64f99b8b-1a33-431b-a06d-b042946dbf35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:49.231783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 655/1327] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:49.515669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143d1938-83f2-41a3-bde4-1e378dec7944 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:49.518012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-143d1938-83f2-41a3-bde4-1e378dec7944 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:49.586384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:49.630670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143d1938-83f2-41a3-bde4-1e378dec7944 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:49.704100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc28e6f2-01b9-4c50-b25b-cf61ce3304e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:49.704714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc28e6f2-01b9-4c50-b25b-cf61ce3304e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:49.704996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc28e6f2-01b9-4c50-b25b-cf61ce3304e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:49.705269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc28e6f2-01b9-4c50-b25b-cf61ce3304e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:49.706868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bc28e6f2-01b9-4c50-b25b-cf61ce3304e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:49.813850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:49.816519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc28e6f2-01b9-4c50-b25b-cf61ce3304e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:49.818048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc28e6f2-01b9-4c50-b25b-cf61ce3304e5 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:49.836383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 673/1328] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:01:52.072069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1219f3-5adc-4288-96e1-f76390941469 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:52.072589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1219f3-5adc-4288-96e1-f76390941469 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:52.072589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1219f3-5adc-4288-96e1-f76390941469 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:52.073942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 656/1329] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:52.080338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0feb3-0857-4004-8868-6f44fa8286d5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:52.080800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0feb3-0857-4004-8868-6f44fa8286d5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:52.080881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0feb3-0857-4004-8868-6f44fa8286d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:52.151385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c5e0feb3-0857-4004-8868-6f44fa8286d5 None None] MFA Rules not processed for user `054bf47407b34e4a875fbfb8ad1c5a1b`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:52.152634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c5e0feb3-0857-4004-8868-6f44fa8286d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:52.248899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 674/1330] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:52 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:52.538453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1219f3-5adc-4288-96e1-f76390941469 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:52.540026 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1a1219f3-5adc-4288-96e1-f76390941469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:52.571946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:52.648130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1219f3-5adc-4288-96e1-f76390941469 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:52.723394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7771988a-7999-4ef7-8e5b-c4671b53d6bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:52.724043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7771988a-7999-4ef7-8e5b-c4671b53d6bc service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:52.724326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7771988a-7999-4ef7-8e5b-c4671b53d6bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:52.724594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7771988a-7999-4ef7-8e5b-c4671b53d6bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:52.726204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7771988a-7999-4ef7-8e5b-c4671b53d6bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:52.807331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:52.842414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7771988a-7999-4ef7-8e5b-c4671b53d6bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:52.846679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7771988a-7999-4ef7-8e5b-c4671b53d6bc service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:52.866878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 657/1331] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:52 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:01:55.451224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a86e01-3a30-48b8-8947-acd9e2b7ad89 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:55.451850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a86e01-3a30-48b8-8947-acd9e2b7ad89 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:55.451850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a86e01-3a30-48b8-8947-acd9e2b7ad89 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:55.453140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 675/1332] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:01:55.458558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c051b-ab99-4618-8d80-9ff752192c78 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:55.458978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c051b-ab99-4618-8d80-9ff752192c78 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:55.459132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25c051b-ab99-4618-8d80-9ff752192c78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:55.524335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f25c051b-ab99-4618-8d80-9ff752192c78 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:55.525540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f25c051b-ab99-4618-8d80-9ff752192c78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:55.621282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 658/1333] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:55.899417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a86e01-3a30-48b8-8947-acd9e2b7ad89 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:55.900994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-10a86e01-3a30-48b8-8947-acd9e2b7ad89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:55.974300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:56.008285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a86e01-3a30-48b8-8947-acd9e2b7ad89 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:56.081605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac3749e-8929-49ef-8538-28f0e5ec9a02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:56.082216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac3749e-8929-49ef-8538-28f0e5ec9a02 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:56.082497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac3749e-8929-49ef-8538-28f0e5ec9a02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:56.082770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac3749e-8929-49ef-8538-28f0e5ec9a02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:56.084391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1ac3749e-8929-49ef-8538-28f0e5ec9a02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:56.191651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac3749e-8929-49ef-8538-28f0e5ec9a02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:56.193052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac3749e-8929-49ef-8538-28f0e5ec9a02 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:56.202809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:56.211537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 676/1334] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:01:58.436876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8711e1-c012-4330-b143-55564f7d1ecd None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:58.437430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8711e1-c012-4330-b143-55564f7d1ecd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:58.437430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8711e1-c012-4330-b143-55564f7d1ecd None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:58.439285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 659/1335] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:01:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:01:58.444752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3dcb75-5e03-4ed4-a206-99dd5d76c4e1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:58.445158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3dcb75-5e03-4ed4-a206-99dd5d76c4e1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:58.445304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3dcb75-5e03-4ed4-a206-99dd5d76c4e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:58.511025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ac3dcb75-5e03-4ed4-a206-99dd5d76c4e1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:01:58.512248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ac3dcb75-5e03-4ed4-a206-99dd5d76c4e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:58.607206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 677/1336] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:01:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:01:58.893222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8711e1-c012-4330-b143-55564f7d1ecd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:01:58.894767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8f8711e1-c012-4330-b143-55564f7d1ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:58.934379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:58.998824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8711e1-c012-4330-b143-55564f7d1ecd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:01:59.072357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf219d0e-0735-43ff-99ff-9aa3cd55efb0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:01:59.072952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf219d0e-0735-43ff-99ff-9aa3cd55efb0 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:01:59.073229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf219d0e-0735-43ff-99ff-9aa3cd55efb0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:01:59.073497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf219d0e-0735-43ff-99ff-9aa3cd55efb0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:01:59.075059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-cf219d0e-0735-43ff-99ff-9aa3cd55efb0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:01:59.159967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:01:59.184800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf219d0e-0735-43ff-99ff-9aa3cd55efb0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:01:59.186480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf219d0e-0735-43ff-99ff-9aa3cd55efb0 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:01:59.205202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 660/1337] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:01:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:01.660221 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2a251-0ecc-4069-8a8a-e6178c2d01e1 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:01.660783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2a251-0ecc-4069-8a8a-e6178c2d01e1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:01.660783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2a251-0ecc-4069-8a8a-e6178c2d01e1 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:01.662006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 678/1338] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:01.667335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fea31a5-5578-4d39-8904-37c30864f03f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:01.667767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fea31a5-5578-4d39-8904-37c30864f03f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:01.667913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fea31a5-5578-4d39-8904-37c30864f03f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:01.732057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9fea31a5-5578-4d39-8904-37c30864f03f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:01.733247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9fea31a5-5578-4d39-8904-37c30864f03f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:01.826614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 661/1339] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:01 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:02.108923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf2a251-0ecc-4069-8a8a-e6178c2d01e1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:02.111059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3cf2a251-0ecc-4069-8a8a-e6178c2d01e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:02.185845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:02.213329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf2a251-0ecc-4069-8a8a-e6178c2d01e1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:02.288243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cf0c5a-6e7b-4a79-ac51-96e26682b4b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:02.288840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cf0c5a-6e7b-4a79-ac51-96e26682b4b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:02.289116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cf0c5a-6e7b-4a79-ac51-96e26682b4b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:02.289384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cf0c5a-6e7b-4a79-ac51-96e26682b4b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:02.290935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c9cf0c5a-6e7b-4a79-ac51-96e26682b4b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:02.399054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9cf0c5a-6e7b-4a79-ac51-96e26682b4b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:02.400455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9cf0c5a-6e7b-4a79-ac51-96e26682b4b3 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:02.418958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 679/1340] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:02 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:06.496869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8834fb-cf90-4039-b633-934f1dd2e9de None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:06.497399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8834fb-cf90-4039-b633-934f1dd2e9de None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:06.497399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8834fb-cf90-4039-b633-934f1dd2e9de None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:06.498754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 662/1341] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:06.504360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2258e0-2220-401d-9561-24374b440b33 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:06.504794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2258e0-2220-401d-9561-24374b440b33 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:06.504915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2258e0-2220-401d-9561-24374b440b33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:06.510543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:06.572127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6f2258e0-2220-401d-9561-24374b440b33 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:06.573243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6f2258e0-2220-401d-9561-24374b440b33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:06.670704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 680/1342] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:06.955326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8834fb-cf90-4039-b633-934f1dd2e9de None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:06.956948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-af8834fb-cf90-4039-b633-934f1dd2e9de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:07.005773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:07.062765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8834fb-cf90-4039-b633-934f1dd2e9de None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:07.136763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc6c578-209b-4f74-a21c-d6914efc6909 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:07.137375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6c578-209b-4f74-a21c-d6914efc6909 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:07.137685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6c578-209b-4f74-a21c-d6914efc6909 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:07.137958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6c578-209b-4f74-a21c-d6914efc6909 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:07.139572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5cc6c578-209b-4f74-a21c-d6914efc6909 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:07.234941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:07.252402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc6c578-209b-4f74-a21c-d6914efc6909 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:07.253845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc6c578-209b-4f74-a21c-d6914efc6909 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:07.272872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 663/1343] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:10.063152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f4447-3ab0-424d-9052-b38a668fa177 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:10.063795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f4447-3ab0-424d-9052-b38a668fa177 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:10.063993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f4447-3ab0-424d-9052-b38a668fa177 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:10.065367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 681/1344] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:10.070380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d9a4f8-7b1c-4f52-998a-08c51dc72ed3 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:10.070793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d9a4f8-7b1c-4f52-998a-08c51dc72ed3 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:10.070937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d9a4f8-7b1c-4f52-998a-08c51dc72ed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:10.139505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-75d9a4f8-7b1c-4f52-998a-08c51dc72ed3 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:10.140746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-75d9a4f8-7b1c-4f52-998a-08c51dc72ed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:10.238636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 664/1345] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:02:10.520167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f4447-3ab0-424d-9052-b38a668fa177 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:10.521819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-214f4447-3ab0-424d-9052-b38a668fa177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:10.612010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:10.631766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f4447-3ab0-424d-9052-b38a668fa177 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:10.709724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1373466-4059-4296-9bf8-4b99d3d64f80 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:10.710336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1373466-4059-4296-9bf8-4b99d3d64f80 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:10.710617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1373466-4059-4296-9bf8-4b99d3d64f80 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:10.710893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1373466-4059-4296-9bf8-4b99d3d64f80 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:10.712518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b1373466-4059-4296-9bf8-4b99d3d64f80 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:10.821644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1373466-4059-4296-9bf8-4b99d3d64f80 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:10.823007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1373466-4059-4296-9bf8-4b99d3d64f80 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:10.841898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 682/1346] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:10 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:14.983544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba87d0-5da6-4f87-a228-79ed85be835c None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:14.984110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba87d0-5da6-4f87-a228-79ed85be835c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:14.984217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba87d0-5da6-4f87-a228-79ed85be835c None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:14.986195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 665/1347] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:14.991342 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f882941-805e-455c-9308-ca2d376f0ffe None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:14.991771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f882941-805e-455c-9308-ca2d376f0ffe None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:14.991935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f882941-805e-455c-9308-ca2d376f0ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:15.007880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:15.059944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1f882941-805e-455c-9308-ca2d376f0ffe None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:15.061424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1f882941-805e-455c-9308-ca2d376f0ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:15.166237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 683/1348] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:15.452340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ba87d0-5da6-4f87-a228-79ed85be835c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:15.454459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-55ba87d0-5da6-4f87-a228-79ed85be835c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:15.511652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:15.556954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ba87d0-5da6-4f87-a228-79ed85be835c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:15.630188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b0facf-54e4-472b-9db0-6f42c7a97d9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:15.630782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0facf-54e4-472b-9db0-6f42c7a97d9b service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:15.631057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0facf-54e4-472b-9db0-6f42c7a97d9b service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:15.631323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0facf-54e4-472b-9db0-6f42c7a97d9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:15.632930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f2b0facf-54e4-472b-9db0-6f42c7a97d9b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:15.751429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:15.755680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b0facf-54e4-472b-9db0-6f42c7a97d9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:15.758951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b0facf-54e4-472b-9db0-6f42c7a97d9b service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:15.900781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 666/1349] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:20.820276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db23ef-b55a-4dbd-930d-2db8219df24e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:20.820738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db23ef-b55a-4dbd-930d-2db8219df24e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:20.821056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db23ef-b55a-4dbd-930d-2db8219df24e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:20.822760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 684/1350] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:02:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:20.829381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68d957-29ca-488f-a636-3442daa17c13 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:20.829840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68d957-29ca-488f-a636-3442daa17c13 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:20.831087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e68d957-29ca-488f-a636-3442daa17c13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:20.922526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9e68d957-29ca-488f-a636-3442daa17c13 None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:20.924213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9e68d957-29ca-488f-a636-3442daa17c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:21.115627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 667/1351] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:02:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:25.164678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=59 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:25.419544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309a2fe-b6ff-4d2f-ae95-ffb68609ff5f None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:25.419863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309a2fe-b6ff-4d2f-ae95-ffb68609ff5f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:25.420419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309a2fe-b6ff-4d2f-ae95-ffb68609ff5f None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:25.421906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 685/1352] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:02:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:25.427503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a63dcc-44be-4a45-8701-1d2acb998434 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:25.428447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a63dcc-44be-4a45-8701-1d2acb998434 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:25.428791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a63dcc-44be-4a45-8701-1d2acb998434 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:25.493849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:25.520564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-04a63dcc-44be-4a45-8701-1d2acb998434 None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:25.521879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-04a63dcc-44be-4a45-8701-1d2acb998434 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:25.639471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 668/1353] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:02:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:26.758432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d05e68-1af9-4b19-8ba2-e63fafe48550 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:26.758972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d05e68-1af9-4b19-8ba2-e63fafe48550 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:26.759091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d05e68-1af9-4b19-8ba2-e63fafe48550 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:26.760568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 686/1354] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:02:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:26.765791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc8bd32-667f-4f6b-b30b-5cd11ffa0b18 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:26.766249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc8bd32-667f-4f6b-b30b-5cd11ffa0b18 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:26.766515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc8bd32-667f-4f6b-b30b-5cd11ffa0b18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:26.843625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-bdc8bd32-667f-4f6b-b30b-5cd11ffa0b18 None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:26.845705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bdc8bd32-667f-4f6b-b30b-5cd11ffa0b18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:26.887390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:26.953381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 669/1355] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:02:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:02:30.570372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b1717a-b48a-459c-a571-1f5dcb5f0f77 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:30.571015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b1717a-b48a-459c-a571-1f5dcb5f0f77 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:30.571015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b1717a-b48a-459c-a571-1f5dcb5f0f77 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:30.572533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 687/1356] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:30.579503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e17b2-f9e7-4411-b29f-1032df3efb09 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:30.580502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e17b2-f9e7-4411-b29f-1032df3efb09 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:30.580766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e17b2-f9e7-4411-b29f-1032df3efb09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:30.668667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-844e17b2-f9e7-4411-b29f-1032df3efb09 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:30.669957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-844e17b2-f9e7-4411-b29f-1032df3efb09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:30.774057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 670/1357] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:31.080442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d98183c-250a-46fb-9944-5c92ee48d32f None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:31.081116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d98183c-250a-46fb-9944-5c92ee48d32f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:31.081638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d98183c-250a-46fb-9944-5c92ee48d32f None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:31.083419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 688/1358] 10.222.0.27 () {54 vars in 962 bytes} [Thu Sep 15 14:02:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:31.092925 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec629169-db0a-4890-8a8d-f39b33ff9edf None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:31.093643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec629169-db0a-4890-8a8d-f39b33ff9edf None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:31.094145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec629169-db0a-4890-8a8d-f39b33ff9edf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:31.114065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:31.186515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ec629169-db0a-4890-8a8d-f39b33ff9edf None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:31.188742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ec629169-db0a-4890-8a8d-f39b33ff9edf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:31.325757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 671/1359] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Sep 15 14:02:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:31.335119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d98183c-250a-46fb-9944-5c92ee48d32f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:31.336747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5d98183c-250a-46fb-9944-5c92ee48d32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:31.496516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d98183c-250a-46fb-9944-5c92ee48d32f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:31.574124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:31.623407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7365a91b-b688-4941-9be0-dfd8383c25d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:31.624338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7365a91b-b688-4941-9be0-dfd8383c25d1 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:31.624626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7365a91b-b688-4941-9be0-dfd8383c25d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:31.624900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7365a91b-b688-4941-9be0-dfd8383c25d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:31.626781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7365a91b-b688-4941-9be0-dfd8383c25d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:31.727614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7365a91b-b688-4941-9be0-dfd8383c25d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:31.729272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7365a91b-b688-4941-9be0-dfd8383c25d1 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:31.748608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 689/1360] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:34.157222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377f67ad-e32e-4150-b1ed-7b54d954f944 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:34.157671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377f67ad-e32e-4150-b1ed-7b54d954f944 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:34.157996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377f67ad-e32e-4150-b1ed-7b54d954f944 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:34.162806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 672/1361] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:34.164868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3dcb45-be0f-4684-a9ab-98592cf1114b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:34.165262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3dcb45-be0f-4684-a9ab-98592cf1114b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:34.166114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3dcb45-be0f-4684-a9ab-98592cf1114b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:34.238647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6b3dcb45-be0f-4684-a9ab-98592cf1114b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:34.239924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6b3dcb45-be0f-4684-a9ab-98592cf1114b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:34.280803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:34.355051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 690/1362] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:34.647771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377f67ad-e32e-4150-b1ed-7b54d954f944 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:34.649527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-377f67ad-e32e-4150-b1ed-7b54d954f944 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:34.727837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:34.835569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377f67ad-e32e-4150-b1ed-7b54d954f944 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:34.971574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff92ed9-88f6-4162-b58a-a178a46a8ed6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:34.972225 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff92ed9-88f6-4162-b58a-a178a46a8ed6 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:34.972500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff92ed9-88f6-4162-b58a-a178a46a8ed6 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:34.972766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff92ed9-88f6-4162-b58a-a178a46a8ed6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:34.974354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5ff92ed9-88f6-4162-b58a-a178a46a8ed6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:34.998388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:35.070107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff92ed9-88f6-4162-b58a-a178a46a8ed6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:35.071716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff92ed9-88f6-4162-b58a-a178a46a8ed6 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:35.090978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 673/1363] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:37.596849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd12b4f-3e50-4c59-bf9b-b219aceb5118 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:37.597457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd12b4f-3e50-4c59-bf9b-b219aceb5118 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:37.597668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd12b4f-3e50-4c59-bf9b-b219aceb5118 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:37.599231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 691/1364] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:37.604746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737ea57d-ea87-4873-98e0-dca731eadae7 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:37.605144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737ea57d-ea87-4873-98e0-dca731eadae7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:37.605318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737ea57d-ea87-4873-98e0-dca731eadae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:37.676785 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-737ea57d-ea87-4873-98e0-dca731eadae7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:37.677987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-737ea57d-ea87-4873-98e0-dca731eadae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:37.785210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:37.792016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 674/1365] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:38.090258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e921053c-58ac-46ca-8197-fcd8631ee546 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:38.090619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e921053c-58ac-46ca-8197-fcd8631ee546 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:38.091091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e921053c-58ac-46ca-8197-fcd8631ee546 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:38.092597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 692/1366] 10.222.0.27 () {54 vars in 962 bytes} [Thu Sep 15 14:02:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:38.098669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c999b9fb-1c31-4fe1-8f32-68fb19b4000e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:38.098955 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c999b9fb-1c31-4fe1-8f32-68fb19b4000e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:38.099242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c999b9fb-1c31-4fe1-8f32-68fb19b4000e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:38.178815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c999b9fb-1c31-4fe1-8f32-68fb19b4000e None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:38.180028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c999b9fb-1c31-4fe1-8f32-68fb19b4000e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:38.307295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 675/1367] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Sep 15 14:02:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:38.317482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e921053c-58ac-46ca-8197-fcd8631ee546 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:38.319162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e921053c-58ac-46ca-8197-fcd8631ee546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:38.483616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e921053c-58ac-46ca-8197-fcd8631ee546 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:38.506927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:38.615551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc9e868-da85-4cf3-9301-cdda7fcba898 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:38.616464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9e868-da85-4cf3-9301-cdda7fcba898 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:38.616464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9e868-da85-4cf3-9301-cdda7fcba898 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:38.616859 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc9e868-da85-4cf3-9301-cdda7fcba898 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:38.618596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2dc9e868-da85-4cf3-9301-cdda7fcba898 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:38.725122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc9e868-da85-4cf3-9301-cdda7fcba898 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:38.726732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc9e868-da85-4cf3-9301-cdda7fcba898 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:38.746675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 693/1368] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:38.776070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbc763-bbd3-4c8e-a26a-c89864636696 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:38.776924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbc763-bbd3-4c8e-a26a-c89864636696 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:38.777197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbc763-bbd3-4c8e-a26a-c89864636696 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:38.779663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 676/1369] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:38.785737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb88609-5deb-44fe-8c71-7b0024991a7b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:38.786024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb88609-5deb-44fe-8c71-7b0024991a7b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:38.786299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb88609-5deb-44fe-8c71-7b0024991a7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:38.820121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:38.867570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7bb88609-5deb-44fe-8c71-7b0024991a7b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:38.869210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7bb88609-5deb-44fe-8c71-7b0024991a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:38.997251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 694/1370] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:39.295884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b1005a-07d2-4711-9056-d9e17bc179bd None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:39.296175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b1005a-07d2-4711-9056-d9e17bc179bd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:39.296451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b1005a-07d2-4711-9056-d9e17bc179bd None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:39.300124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 677/1371] 10.222.0.27 () {54 vars in 964 bytes} [Thu Sep 15 14:02:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:39.306133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a822c6-dcd4-4259-904b-bd4bbebbc407 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:39.306520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a822c6-dcd4-4259-904b-bd4bbebbc407 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:39.306892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a822c6-dcd4-4259-904b-bd4bbebbc407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:39.392238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-25a822c6-dcd4-4259-904b-bd4bbebbc407 None None] MFA Rules not processed for user `f54979e322484a42bb57d1b3a619652c`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:39.393448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-25a822c6-dcd4-4259-904b-bd4bbebbc407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:39.448060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:39.478796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 695/1372] 10.222.0.27 () {58 vars in 1061 bytes} [Thu Sep 15 14:02:39 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:39.486359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b1005a-07d2-4711-9056-d9e17bc179bd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:39.488047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-60b1005a-07d2-4711-9056-d9e17bc179bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:39.543499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:39.602183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b1005a-07d2-4711-9056-d9e17bc179bd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:39.690650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d212acd-9f4c-495a-b07a-ed85bf89ef10 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:39.691416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d212acd-9f4c-495a-b07a-ed85bf89ef10 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:39.691802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d212acd-9f4c-495a-b07a-ed85bf89ef10 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:39.691921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d212acd-9f4c-495a-b07a-ed85bf89ef10 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:39.695662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0d212acd-9f4c-495a-b07a-ed85bf89ef10 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:39.853748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:39.871782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d212acd-9f4c-495a-b07a-ed85bf89ef10 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:39.873446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d212acd-9f4c-495a-b07a-ed85bf89ef10 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:39.900092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 678/1373] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:02:40.997834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a01bc6-c399-417f-adf8-f28ab6d02069 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:40.998402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a01bc6-c399-417f-adf8-f28ab6d02069 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:40.998402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a01bc6-c399-417f-adf8-f28ab6d02069 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:40.999846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 696/1374] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:41.006019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49bcea7-ff23-412c-b6fb-1e6875b89148 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:41.006446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49bcea7-ff23-412c-b6fb-1e6875b89148 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:41.006544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49bcea7-ff23-412c-b6fb-1e6875b89148 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:41.072704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f49bcea7-ff23-412c-b6fb-1e6875b89148 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:41.073904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f49bcea7-ff23-412c-b6fb-1e6875b89148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:41.181488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 679/1375] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:02:41.478468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a01bc6-c399-417f-adf8-f28ab6d02069 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:41.480129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-24a01bc6-c399-417f-adf8-f28ab6d02069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:41.637748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a01bc6-c399-417f-adf8-f28ab6d02069 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:41.691280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:41.754411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d263774-d31e-4f20-9d56-08f594621da1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:41.755031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d263774-d31e-4f20-9d56-08f594621da1 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:41.755339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d263774-d31e-4f20-9d56-08f594621da1 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:41.755615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d263774-d31e-4f20-9d56-08f594621da1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:41.757456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4d263774-d31e-4f20-9d56-08f594621da1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:41.852554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d263774-d31e-4f20-9d56-08f594621da1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:41.854162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d263774-d31e-4f20-9d56-08f594621da1 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:41.873693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 697/1376] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:44.233818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5650311f-f94e-4697-8e28-9d830be9e922 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:44.234387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5650311f-f94e-4697-8e28-9d830be9e922 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:44.234387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5650311f-f94e-4697-8e28-9d830be9e922 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:44.236962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 680/1377] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:44.242236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9931533-77cf-4b68-9c48-7374417434de None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:44.242663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9931533-77cf-4b68-9c48-7374417434de None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:44.242792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9931533-77cf-4b68-9c48-7374417434de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:44.318722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e9931533-77cf-4b68-9c48-7374417434de None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:44.319991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e9931533-77cf-4b68-9c48-7374417434de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:44.328422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:44.432376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 698/1378] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:44.727342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5650311f-f94e-4697-8e28-9d830be9e922 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:44.729148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5650311f-f94e-4697-8e28-9d830be9e922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:44.792148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:44.880282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5650311f-f94e-4697-8e28-9d830be9e922 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:44.998887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd24325-2ef7-4670-8ff4-406a7ed754af service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:44.999401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd24325-2ef7-4670-8ff4-406a7ed754af service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:44.999676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd24325-2ef7-4670-8ff4-406a7ed754af service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:45.000018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd24325-2ef7-4670-8ff4-406a7ed754af service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:45.001955 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1fd24325-2ef7-4670-8ff4-406a7ed754af service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:45.022645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:45.102269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd24325-2ef7-4670-8ff4-406a7ed754af service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:45.104028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd24325-2ef7-4670-8ff4-406a7ed754af service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:45.123265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 681/1379] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:47.346569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ad53ea-5d85-4a1c-a52b-9c6379bac4c2 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:47.347085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ad53ea-5d85-4a1c-a52b-9c6379bac4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:47.347207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ad53ea-5d85-4a1c-a52b-9c6379bac4c2 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:47.348649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 699/1380] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:47.353807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847dc7f4-8c87-4bd6-8531-893d43b9cb6a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:47.354208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847dc7f4-8c87-4bd6-8531-893d43b9cb6a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:47.354387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847dc7f4-8c87-4bd6-8531-893d43b9cb6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:47.425152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-847dc7f4-8c87-4bd6-8531-893d43b9cb6a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:47.426490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-847dc7f4-8c87-4bd6-8531-893d43b9cb6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:47.584379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:47.591447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 682/1381] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:47.902150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ad53ea-5d85-4a1c-a52b-9c6379bac4c2 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:47.903679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-94ad53ea-5d85-4a1c-a52b-9c6379bac4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:48.050449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:48.061322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ad53ea-5d85-4a1c-a52b-9c6379bac4c2 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:48.176031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408b64ee-45d7-46a1-a0e2-13579450bc2d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:48.176516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408b64ee-45d7-46a1-a0e2-13579450bc2d service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:48.176800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408b64ee-45d7-46a1-a0e2-13579450bc2d service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:48.177076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408b64ee-45d7-46a1-a0e2-13579450bc2d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:48.178729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-408b64ee-45d7-46a1-a0e2-13579450bc2d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:48.271435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408b64ee-45d7-46a1-a0e2-13579450bc2d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:48.273090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408b64ee-45d7-46a1-a0e2-13579450bc2d service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:48.283427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:48.292718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 700/1382] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:50.686171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a5a1cd-e2c4-40b1-8a08-06b8fbe22eee None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:50.686685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a5a1cd-e2c4-40b1-8a08-06b8fbe22eee None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:50.686685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a5a1cd-e2c4-40b1-8a08-06b8fbe22eee None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:50.688859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 683/1383] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:02:50.694746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f499ca-7ab3-41b8-b324-1f1d21c20818 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:50.695350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f499ca-7ab3-41b8-b324-1f1d21c20818 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:50.695350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f499ca-7ab3-41b8-b324-1f1d21c20818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:50.771007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-83f499ca-7ab3-41b8-b324-1f1d21c20818 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:50.772258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-83f499ca-7ab3-41b8-b324-1f1d21c20818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:50.877599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 701/1384] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:51.167062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a5a1cd-e2c4-40b1-8a08-06b8fbe22eee None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:51.168625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d2a5a1cd-e2c4-40b1-8a08-06b8fbe22eee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:51.331845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a5a1cd-e2c4-40b1-8a08-06b8fbe22eee None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:51.409562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:51.458591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2568d4c-0470-4388-a84d-bdf518d9152a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:51.459198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2568d4c-0470-4388-a84d-bdf518d9152a service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:51.459478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2568d4c-0470-4388-a84d-bdf518d9152a service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:51.459774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2568d4c-0470-4388-a84d-bdf518d9152a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:51.461365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e2568d4c-0470-4388-a84d-bdf518d9152a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:51.555891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2568d4c-0470-4388-a84d-bdf518d9152a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:51.557490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2568d4c-0470-4388-a84d-bdf518d9152a service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:51.575694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 684/1385] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:53.760536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98f30b6-fff2-4b47-bfc4-96b88bfde011 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:53.761096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98f30b6-fff2-4b47-bfc4-96b88bfde011 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:53.761228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98f30b6-fff2-4b47-bfc4-96b88bfde011 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:53.762930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 702/1386] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:53.768806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7678163-7c8e-4992-bc72-ffc8100603d7 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:53.769266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7678163-7c8e-4992-bc72-ffc8100603d7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:53.769364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7678163-7c8e-4992-bc72-ffc8100603d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:53.841969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c7678163-7c8e-4992-bc72-ffc8100603d7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:53.843293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c7678163-7c8e-4992-bc72-ffc8100603d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:53.881650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:53.955707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 685/1387] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:54.256308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98f30b6-fff2-4b47-bfc4-96b88bfde011 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:54.257902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c98f30b6-fff2-4b47-bfc4-96b88bfde011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:54.341169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:54.414870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98f30b6-fff2-4b47-bfc4-96b88bfde011 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:54.534276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab8aa6e-1bee-4cee-a346-542575fe1024 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:54.534837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab8aa6e-1bee-4cee-a346-542575fe1024 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:54.535099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab8aa6e-1bee-4cee-a346-542575fe1024 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:54.535376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab8aa6e-1bee-4cee-a346-542575fe1024 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:54.537033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1ab8aa6e-1bee-4cee-a346-542575fe1024 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:54.582569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:54.635707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab8aa6e-1bee-4cee-a346-542575fe1024 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:54.637334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab8aa6e-1bee-4cee-a346-542575fe1024 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:54.655948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 703/1388] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:02:56.938574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f15d6-4bda-4d54-8324-4ed082e802bc None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:56.939084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f15d6-4bda-4d54-8324-4ed082e802bc None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:56.939084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f15d6-4bda-4d54-8324-4ed082e802bc None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:56.940537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 686/1389] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:02:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:02:56.946135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f040c-a8a1-4bfa-a93c-125089b2b3cb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:56.946650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f040c-a8a1-4bfa-a93c-125089b2b3cb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:56.946824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f040c-a8a1-4bfa-a93c-125089b2b3cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:57.023911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-177f040c-a8a1-4bfa-a93c-125089b2b3cb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:02:57.025484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-177f040c-a8a1-4bfa-a93c-125089b2b3cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:57.135396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 704/1390] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:02:57.427764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044f15d6-4bda-4d54-8324-4ed082e802bc None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:02:57.429293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-044f15d6-4bda-4d54-8324-4ed082e802bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:57.585514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044f15d6-4bda-4d54-8324-4ed082e802bc None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:02:57.602547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:02:57.709106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e353b1a-9a16-4ffb-9765-30f28f3d6073 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:02:57.709721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e353b1a-9a16-4ffb-9765-30f28f3d6073 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:02:57.710001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e353b1a-9a16-4ffb-9765-30f28f3d6073 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:02:57.710271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e353b1a-9a16-4ffb-9765-30f28f3d6073 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:02:57.711967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3e353b1a-9a16-4ffb-9765-30f28f3d6073 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:02:57.810728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e353b1a-9a16-4ffb-9765-30f28f3d6073 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:02:57.814525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e353b1a-9a16-4ffb-9765-30f28f3d6073 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:02:57.834890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 687/1391] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:02:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:00.035071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e19f8-3db9-4bf2-900b-a6891f739c78 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:00.035633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e19f8-3db9-4bf2-900b-a6891f739c78 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:00.035633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e19f8-3db9-4bf2-900b-a6891f739c78 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:00.037009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 705/1392] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:03:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:00.042378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a45e0b-4d6d-4609-b9af-20cb99637d6b None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:00.042805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a45e0b-4d6d-4609-b9af-20cb99637d6b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:00.043055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a45e0b-4d6d-4609-b9af-20cb99637d6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:00.069039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:00.111506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b3a45e0b-4d6d-4609-b9af-20cb99637d6b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:00.112752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b3a45e0b-4d6d-4609-b9af-20cb99637d6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:00.214471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 688/1393] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:03:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:00.244562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a928f4-c425-4fbe-8247-63cb7c9fc221 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:00.244938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a928f4-c425-4fbe-8247-63cb7c9fc221 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:00.245381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a928f4-c425-4fbe-8247-63cb7c9fc221 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:00.246779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 706/1394] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:03:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:00.252220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042db05e-c88f-4b3c-be4a-8f13ea8985b2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:00.252505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042db05e-c88f-4b3c-be4a-8f13ea8985b2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:00.252779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042db05e-c88f-4b3c-be4a-8f13ea8985b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:00.322182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-042db05e-c88f-4b3c-be4a-8f13ea8985b2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:00.323379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-042db05e-c88f-4b3c-be4a-8f13ea8985b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:00.376113 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:00.424581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 689/1395] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:03:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:00.509783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a928f4-c425-4fbe-8247-63cb7c9fc221 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:00.511799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b5a928f4-c425-4fbe-8247-63cb7c9fc221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:00.525725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:00.664611 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a928f4-c425-4fbe-8247-63cb7c9fc221 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:00.730361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2136c-c0a0-4e92-9b24-671d336079d7 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:00.730777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2136c-c0a0-4e92-9b24-671d336079d7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:00.731121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2136c-c0a0-4e92-9b24-671d336079d7 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:00.732570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 690/1396] 10.222.0.27 () {54 vars in 962 bytes} [Thu Sep 15 14:03:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:03:00.738399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72bc32d-5931-4eaf-86e0-1d4401e7b0f4 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:00.738758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72bc32d-5931-4eaf-86e0-1d4401e7b0f4 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:00.739093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72bc32d-5931-4eaf-86e0-1d4401e7b0f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:00.757236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:00.786571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba3f8d7-3d1a-49cd-be84-1b1dfa653877 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:00.787190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3f8d7-3d1a-49cd-be84-1b1dfa653877 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:00.787471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3f8d7-3d1a-49cd-be84-1b1dfa653877 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:00.787774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3f8d7-3d1a-49cd-be84-1b1dfa653877 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:00.789392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-fba3f8d7-3d1a-49cd-be84-1b1dfa653877 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:00.818389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f72bc32d-5931-4eaf-86e0-1d4401e7b0f4 None None] MFA Rules not processed for user `54db2e6805fd40f79c03eb3b2319f2b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:00.819591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f72bc32d-5931-4eaf-86e0-1d4401e7b0f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:00.883325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba3f8d7-3d1a-49cd-be84-1b1dfa653877 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:00.884997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba3f8d7-3d1a-49cd-be84-1b1dfa653877 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:00.898876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 691/1397] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Sep 15 14:03:00 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:03:00.903712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 707/1398] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:00.908375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72bc32d-5931-4eaf-86e0-1d4401e7b0f4 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:00.916074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72bc32d-5931-4eaf-86e0-1d4401e7b0f4 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:00.964447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:01.005446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05128a00-c08a-4f1d-be63-dd53e49f1b91 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '54db2e6805fd40f79c03eb3b2319f2b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:01.006068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05128a00-c08a-4f1d-be63-dd53e49f1b91 service glance] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:01.006347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05128a00-c08a-4f1d-be63-dd53e49f1b91 service glance] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:01.006616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05128a00-c08a-4f1d-be63-dd53e49f1b91 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:01.008349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-05128a00-c08a-4f1d-be63-dd53e49f1b91 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:01.133288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05128a00-c08a-4f1d-be63-dd53e49f1b91 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:01.134732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05128a00-c08a-4f1d-be63-dd53e49f1b91 service glance] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:01.155572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 692/1399] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:03:01.911854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4a9d4-c7bb-4647-bba8-4f6fac040d3d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:01.912351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4a9d4-c7bb-4647-bba8-4f6fac040d3d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:01.912458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4a9d4-c7bb-4647-bba8-4f6fac040d3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:01.986640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a7d4a9d4-c7bb-4647-bba8-4f6fac040d3d None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:01.987891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a7d4a9d4-c7bb-4647-bba8-4f6fac040d3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:02.038252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:02.077167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 708/1400] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:03:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:02.118831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4fc439-60ac-4673-9ce3-eaf349480d12 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:02.119188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4fc439-60ac-4673-9ce3-eaf349480d12 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:02.119535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4fc439-60ac-4673-9ce3-eaf349480d12 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:02.121173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 693/1401] 10.222.0.27 () {54 vars in 943 bytes} [Thu Sep 15 14:03:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:02.127927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f5f96-ed36-4d98-91cf-5d0c47a72d3f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:02.128218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f5f96-ed36-4d98-91cf-5d0c47a72d3f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:02.128496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f5f96-ed36-4d98-91cf-5d0c47a72d3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:02.206332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-169f5f96-ed36-4d98-91cf-5d0c47a72d3f None None] MFA Rules not processed for user `30fc2a26b4fc4d24b984fa8097752055`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:02.207524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-169f5f96-ed36-4d98-91cf-5d0c47a72d3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:02.324537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 709/1402] 10.222.0.27 () {58 vars in 1040 bytes} [Thu Sep 15 14:03:02 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:03:02.334828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4fc439-60ac-4673-9ce3-eaf349480d12 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:02.336455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0e4fc439-60ac-4673-9ce3-eaf349480d12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:02.411215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:02.476558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4fc439-60ac-4673-9ce3-eaf349480d12 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:02.583963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff46238-a7f5-4bf2-b8e1-63cf07bb49d7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:02.584577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff46238-a7f5-4bf2-b8e1-63cf07bb49d7 service swift] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:02.584856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff46238-a7f5-4bf2-b8e1-63cf07bb49d7 service swift] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:02.585127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff46238-a7f5-4bf2-b8e1-63cf07bb49d7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:02.586816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7ff46238-a7f5-4bf2-b8e1-63cf07bb49d7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:02.644654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:02.673697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff46238-a7f5-4bf2-b8e1-63cf07bb49d7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:02.675293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff46238-a7f5-4bf2-b8e1-63cf07bb49d7 service swift] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:02.683371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 694/1403] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:03:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:03:03.128458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f80dd-cfee-489a-a76b-a05514bf3bc0 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:03.129034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f80dd-cfee-489a-a76b-a05514bf3bc0 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:03.129435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f80dd-cfee-489a-a76b-a05514bf3bc0 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:03.133040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 710/1404] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:03:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:03.136953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f93c520-0162-45c7-a5f3-0696a540b2aa None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:03.137594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f93c520-0162-45c7-a5f3-0696a540b2aa None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:03.137938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f93c520-0162-45c7-a5f3-0696a540b2aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:03.214895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6f93c520-0162-45c7-a5f3-0696a540b2aa None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:03.216446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6f93c520-0162-45c7-a5f3-0696a540b2aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:03.337331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 695/1405] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:03:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:03.632017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61f80dd-cfee-489a-a76b-a05514bf3bc0 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:03.633673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e61f80dd-cfee-489a-a76b-a05514bf3bc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:03.665400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:03.814912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61f80dd-cfee-489a-a76b-a05514bf3bc0 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:03.920756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:03.945079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1289a17-183a-4abf-9642-e7459554e583 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:03.945825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1289a17-183a-4abf-9642-e7459554e583 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:03.946113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1289a17-183a-4abf-9642-e7459554e583 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:03.946412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1289a17-183a-4abf-9642-e7459554e583 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:03.948302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c1289a17-183a-4abf-9642-e7459554e583 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:04.053856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1289a17-183a-4abf-9642-e7459554e583 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:04.055632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1289a17-183a-4abf-9642-e7459554e583 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:04.079923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 711/1406] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:06.337669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391da875-fe61-4c6c-8886-207158805228 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:06.338228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391da875-fe61-4c6c-8886-207158805228 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:06.338228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391da875-fe61-4c6c-8886-207158805228 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:06.339556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 696/1407] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:03:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:06.345654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca90eac-52f6-4df9-9804-91dabd53af4b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:06.346114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca90eac-52f6-4df9-9804-91dabd53af4b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:06.346328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca90eac-52f6-4df9-9804-91dabd53af4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:06.421813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2ca90eac-52f6-4df9-9804-91dabd53af4b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:06.423129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2ca90eac-52f6-4df9-9804-91dabd53af4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:06.486390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:06.538348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 712/1408] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:03:06.835757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391da875-fe61-4c6c-8886-207158805228 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:06.837282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-391da875-fe61-4c6c-8886-207158805228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:06.894017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:07.032658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391da875-fe61-4c6c-8886-207158805228 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:07.197986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e45c887-27c1-48fb-9382-aed286738077 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:07.198599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45c887-27c1-48fb-9382-aed286738077 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:07.198901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45c887-27c1-48fb-9382-aed286738077 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:07.199170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45c887-27c1-48fb-9382-aed286738077 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:07.200800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6e45c887-27c1-48fb-9382-aed286738077 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:07.213506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:07.305530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e45c887-27c1-48fb-9382-aed286738077 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:07.307168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e45c887-27c1-48fb-9382-aed286738077 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:07.340112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 697/1409] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:16.536289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=17 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:17.329309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=41 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:25.378835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fcd034-a6b3-4bed-8370-a80e432e21a6 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:25.379367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fcd034-a6b3-4bed-8370-a80e432e21a6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:25.379367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fcd034-a6b3-4bed-8370-a80e432e21a6 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:25.381017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 713/1410] 10.222.0.27 () {54 vars in 918 bytes} [Thu Sep 15 14:03:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:25.385779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4f44a-1b6f-412a-8177-a8f1bc9a6df0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:25.386222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4f44a-1b6f-412a-8177-a8f1bc9a6df0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:25.386348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa4f44a-1b6f-412a-8177-a8f1bc9a6df0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:25.461078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2fa4f44a-1b6f-412a-8177-a8f1bc9a6df0 None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:25.462268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2fa4f44a-1b6f-412a-8177-a8f1bc9a6df0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:25.568559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 698/1411] 10.222.0.27 () {58 vars in 1015 bytes} [Thu Sep 15 14:03:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:25.583493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab77ad1d-6022-4020-b582-f2c094ce3909 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:25.583815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab77ad1d-6022-4020-b582-f2c094ce3909 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:25.584092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab77ad1d-6022-4020-b582-f2c094ce3909 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:25.585396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 714/1412] 10.222.0.27 () {54 vars in 918 bytes} [Thu Sep 15 14:03:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:25.590015 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469df5db-4037-43a5-a4f6-5fb883d7968f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:25.590295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469df5db-4037-43a5-a4f6-5fb883d7968f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:25.590564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469df5db-4037-43a5-a4f6-5fb883d7968f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:25.665700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:25.668252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-469df5db-4037-43a5-a4f6-5fb883d7968f None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:25.670288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-469df5db-4037-43a5-a4f6-5fb883d7968f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:25.784987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 699/1413] 10.222.0.27 () {58 vars in 1015 bytes} [Thu Sep 15 14:03:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:28.366368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b981ee8-14ea-4816-9c24-c19c45091087 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:28.366800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b981ee8-14ea-4816-9c24-c19c45091087 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:28.366914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b981ee8-14ea-4816-9c24-c19c45091087 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:28.368283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 715/1414] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:03:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:28.374143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce4f20-a106-4e04-b382-1b3008d66319 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:28.374597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce4f20-a106-4e04-b382-1b3008d66319 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:28.374705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ce4f20-a106-4e04-b382-1b3008d66319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:28.440805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-25ce4f20-a106-4e04-b382-1b3008d66319 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:28.442006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-25ce4f20-a106-4e04-b382-1b3008d66319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:28.498460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:28.539126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 700/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:03:28.829155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b981ee8-14ea-4816-9c24-c19c45091087 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:28.830669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1b981ee8-14ea-4816-9c24-c19c45091087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:28.982671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b981ee8-14ea-4816-9c24-c19c45091087 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:29.059278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:29.100237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c71767e-07de-4e74-bb2d-4639fcc31147 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:29.100771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c71767e-07de-4e74-bb2d-4639fcc31147 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:29.101052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c71767e-07de-4e74-bb2d-4639fcc31147 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:29.101326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c71767e-07de-4e74-bb2d-4639fcc31147 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:29.102928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1c71767e-07de-4e74-bb2d-4639fcc31147 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:29.192871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c71767e-07de-4e74-bb2d-4639fcc31147 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:29.194473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c71767e-07de-4e74-bb2d-4639fcc31147 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:29.213339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 716/1416] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:30.358439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6146c6b4-b4b7-45ff-9970-a4f75878afee None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:30.358924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6146c6b4-b4b7-45ff-9970-a4f75878afee None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:30.359028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6146c6b4-b4b7-45ff-9970-a4f75878afee None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:30.360409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 701/1417] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:03:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:30.366689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4914c7-52e0-44f6-984d-87d0b392a62d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:30.367129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4914c7-52e0-44f6-984d-87d0b392a62d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:30.367236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4914c7-52e0-44f6-984d-87d0b392a62d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:30.443023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1e4914c7-52e0-44f6-984d-87d0b392a62d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:30.444241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1e4914c7-52e0-44f6-984d-87d0b392a62d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:30.471288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:30.541637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 717/1418] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:03:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:30.693576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6146c6b4-b4b7-45ff-9970-a4f75878afee None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:30.695093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6146c6b4-b4b7-45ff-9970-a4f75878afee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:30.842454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6146c6b4-b4b7-45ff-9970-a4f75878afee None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:30.881550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:30.957317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f44bbb-80cb-4d0a-8b42-260002b4f911 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:30.957919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f44bbb-80cb-4d0a-8b42-260002b4f911 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:30.958191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f44bbb-80cb-4d0a-8b42-260002b4f911 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:30.958454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f44bbb-80cb-4d0a-8b42-260002b4f911 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:30.960059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-72f44bbb-80cb-4d0a-8b42-260002b4f911 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:31.049227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f44bbb-80cb-4d0a-8b42-260002b4f911 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:31.050800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f44bbb-80cb-4d0a-8b42-260002b4f911 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:31.068780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 702/1419] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:32.203637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e7999-722a-42d3-96de-125f5413ed30 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:32.204162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e7999-722a-42d3-96de-125f5413ed30 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:32.204162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e7999-722a-42d3-96de-125f5413ed30 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:32.205448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 718/1420] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:03:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:32.211883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72905134-e27e-4899-ab57-140c2dea9be6 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:32.212322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72905134-e27e-4899-ab57-140c2dea9be6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:32.212430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72905134-e27e-4899-ab57-140c2dea9be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:32.266012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:32.288133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-72905134-e27e-4899-ab57-140c2dea9be6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:32.289337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-72905134-e27e-4899-ab57-140c2dea9be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:32.388661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 703/1421] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:03:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:32.542603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63e7999-722a-42d3-96de-125f5413ed30 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:32.544678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b63e7999-722a-42d3-96de-125f5413ed30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:32.688380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63e7999-722a-42d3-96de-125f5413ed30 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:32.700237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:32.800400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56c349f-a9b1-453d-b53b-8a1c0ff29ded service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:32.800916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c349f-a9b1-453d-b53b-8a1c0ff29ded service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:32.801194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c349f-a9b1-453d-b53b-8a1c0ff29ded service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:32.801463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c349f-a9b1-453d-b53b-8a1c0ff29ded service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:32.803033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a56c349f-a9b1-453d-b53b-8a1c0ff29ded service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:32.891176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56c349f-a9b1-453d-b53b-8a1c0ff29ded service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:32.892789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56c349f-a9b1-453d-b53b-8a1c0ff29ded service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:32.911151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 719/1422] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:34.045161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3abad-db31-42b6-bcc0-013fe67e528f None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:34.045734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3abad-db31-42b6-bcc0-013fe67e528f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:34.045835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3abad-db31-42b6-bcc0-013fe67e528f None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:34.047171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 704/1423] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:03:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:34.053541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d04574-cec6-432f-b2e0-cb6ef5bc7210 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:34.053981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d04574-cec6-432f-b2e0-cb6ef5bc7210 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:34.054085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d04574-cec6-432f-b2e0-cb6ef5bc7210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:34.074150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:34.131262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-41d04574-cec6-432f-b2e0-cb6ef5bc7210 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:34.132493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-41d04574-cec6-432f-b2e0-cb6ef5bc7210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:34.227253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 720/1424] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:03:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:34.378928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d3abad-db31-42b6-bcc0-013fe67e528f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:34.380481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-42d3abad-db31-42b6-bcc0-013fe67e528f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:34.489966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:34.526042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d3abad-db31-42b6-bcc0-013fe67e528f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:34.638023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab398514-5eb1-46bd-8a68-0d98c1fcf9b5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:34.638553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab398514-5eb1-46bd-8a68-0d98c1fcf9b5 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:34.638828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab398514-5eb1-46bd-8a68-0d98c1fcf9b5 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:34.639098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab398514-5eb1-46bd-8a68-0d98c1fcf9b5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:34.640707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ab398514-5eb1-46bd-8a68-0d98c1fcf9b5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:34.716689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:34.729941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab398514-5eb1-46bd-8a68-0d98c1fcf9b5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:34.731751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab398514-5eb1-46bd-8a68-0d98c1fcf9b5 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:34.749437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 705/1425] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:35.855973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fdce2a-c3c4-4821-abce-74f33ed349b7 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:35.857198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fdce2a-c3c4-4821-abce-74f33ed349b7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:35.857198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fdce2a-c3c4-4821-abce-74f33ed349b7 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:35.858508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 721/1426] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:03:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:35.864672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9aa0ab-7b5f-43ba-8587-18b8a1349ca9 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:35.865091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9aa0ab-7b5f-43ba-8587-18b8a1349ca9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:35.865226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9aa0ab-7b5f-43ba-8587-18b8a1349ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:35.940070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2a9aa0ab-7b5f-43ba-8587-18b8a1349ca9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:35.941196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2a9aa0ab-7b5f-43ba-8587-18b8a1349ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:36.036124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 706/1427] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:03:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:36.187642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fdce2a-c3c4-4821-abce-74f33ed349b7 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:36.189246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-74fdce2a-c3c4-4821-abce-74f33ed349b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:36.271217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:36.334355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fdce2a-c3c4-4821-abce-74f33ed349b7 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:36.446338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196b4c3e-b443-4831-8119-e95f3a560614 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:36.446951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196b4c3e-b443-4831-8119-e95f3a560614 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:36.447256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196b4c3e-b443-4831-8119-e95f3a560614 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:36.447526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196b4c3e-b443-4831-8119-e95f3a560614 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:36.449165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-196b4c3e-b443-4831-8119-e95f3a560614 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:36.497836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:36.539470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196b4c3e-b443-4831-8119-e95f3a560614 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:36.541014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196b4c3e-b443-4831-8119-e95f3a560614 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:36.561315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 722/1428] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:37.618328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09151e25-d4a9-48eb-900f-42c9ea771664 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:37.618903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09151e25-d4a9-48eb-900f-42c9ea771664 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:37.618903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09151e25-d4a9-48eb-900f-42c9ea771664 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:37.620253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 707/1429] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:03:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:37.627372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1d1cdd-9c29-401e-bab5-45ea61c4fdbf None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:37.627821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1d1cdd-9c29-401e-bab5-45ea61c4fdbf None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:37.627961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1d1cdd-9c29-401e-bab5-45ea61c4fdbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:37.703382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-4b1d1cdd-9c29-401e-bab5-45ea61c4fdbf None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:37.704625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4b1d1cdd-9c29-401e-bab5-45ea61c4fdbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:37.813162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 723/1430] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:03:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:37.962967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09151e25-d4a9-48eb-900f-42c9ea771664 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:37.964526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-09151e25-d4a9-48eb-900f-42c9ea771664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:38.013666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:38.110598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09151e25-d4a9-48eb-900f-42c9ea771664 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:38.226150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf8c31d-46e2-4ad5-bb7d-ec145ee18684 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:38.226666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf8c31d-46e2-4ad5-bb7d-ec145ee18684 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:38.226940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf8c31d-46e2-4ad5-bb7d-ec145ee18684 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:38.227208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf8c31d-46e2-4ad5-bb7d-ec145ee18684 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:38.228818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-baf8c31d-46e2-4ad5-bb7d-ec145ee18684 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:38.236416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:38.319714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf8c31d-46e2-4ad5-bb7d-ec145ee18684 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:38.321303 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf8c31d-46e2-4ad5-bb7d-ec145ee18684 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:38.339532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 708/1431] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:39.400950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca06dc-318e-4c25-b833-c6ee7bc045b3 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:39.401511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca06dc-318e-4c25-b833-c6ee7bc045b3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:39.401511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca06dc-318e-4c25-b833-c6ee7bc045b3 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:39.402763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 724/1432] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:03:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:39.409234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7580d4cb-0910-4344-8ba7-e8a3bbe85bba None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:39.409740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7580d4cb-0910-4344-8ba7-e8a3bbe85bba None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:39.409846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7580d4cb-0910-4344-8ba7-e8a3bbe85bba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:39.485122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-7580d4cb-0910-4344-8ba7-e8a3bbe85bba None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:39.486320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7580d4cb-0910-4344-8ba7-e8a3bbe85bba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:39.553399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:39.583399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 709/1433] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:03:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:03:39.735456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ca06dc-318e-4c25-b833-c6ee7bc045b3 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:39.737023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-61ca06dc-318e-4c25-b833-c6ee7bc045b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:39.754905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:39.881538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ca06dc-318e-4c25-b833-c6ee7bc045b3 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:39.968413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:39.993947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abd39e4-1d7f-4875-8336-9e24a487ed6f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:39.994487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abd39e4-1d7f-4875-8336-9e24a487ed6f service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:39.994766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abd39e4-1d7f-4875-8336-9e24a487ed6f service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:39.995040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abd39e4-1d7f-4875-8336-9e24a487ed6f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:39.997304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6abd39e4-1d7f-4875-8336-9e24a487ed6f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:40.084975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abd39e4-1d7f-4875-8336-9e24a487ed6f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:40.086572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abd39e4-1d7f-4875-8336-9e24a487ed6f service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:40.104706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 725/1434] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:41.161435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115f400-e279-4a68-8f06-1307a861ee65 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:41.162009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115f400-e279-4a68-8f06-1307a861ee65 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:41.162009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115f400-e279-4a68-8f06-1307a861ee65 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:41.163336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 710/1435] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:03:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:41.169955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13683171-d7c5-4276-82b2-cd25350ad51f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:41.170410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13683171-d7c5-4276-82b2-cd25350ad51f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:41.170525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13683171-d7c5-4276-82b2-cd25350ad51f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:41.247999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-13683171-d7c5-4276-82b2-cd25350ad51f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:41.249202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-13683171-d7c5-4276-82b2-cd25350ad51f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:41.290681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:41.345886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 726/1436] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:03:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:41.497381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5115f400-e279-4a68-8f06-1307a861ee65 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:41.498904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5115f400-e279-4a68-8f06-1307a861ee65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:41.646113 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5115f400-e279-4a68-8f06-1307a861ee65 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:41.699308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:41.757725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d79209b-046e-4528-b466-9156847ac47e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:41.758253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d79209b-046e-4528-b466-9156847ac47e service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:41.758528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d79209b-046e-4528-b466-9156847ac47e service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:41.758796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d79209b-046e-4528-b466-9156847ac47e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:41.760401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5d79209b-046e-4528-b466-9156847ac47e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:41.847491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d79209b-046e-4528-b466-9156847ac47e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:41.849226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d79209b-046e-4528-b466-9156847ac47e service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:41.866532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 711/1437] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:42.926105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c097b0ff-f69d-48ad-9ae2-91f69efaafe5 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:42.926687 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c097b0ff-f69d-48ad-9ae2-91f69efaafe5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:42.926687 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c097b0ff-f69d-48ad-9ae2-91f69efaafe5 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:42.928021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 727/1438] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:03:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:42.935111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29705364-56e6-40e8-972c-318d2b97e6f2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:42.935610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29705364-56e6-40e8-972c-318d2b97e6f2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:42.935768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29705364-56e6-40e8-972c-318d2b97e6f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:43.025133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-29705364-56e6-40e8-972c-318d2b97e6f2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:43.026446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-29705364-56e6-40e8-972c-318d2b97e6f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:43.029136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:43.129951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 712/1439] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:03:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:43.287834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c097b0ff-f69d-48ad-9ae2-91f69efaafe5 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:43.289752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c097b0ff-f69d-48ad-9ae2-91f69efaafe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:43.448591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c097b0ff-f69d-48ad-9ae2-91f69efaafe5 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:43.481992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:43.586893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec86cae-a4dc-4133-b34d-e19ddc8a244f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:43.587800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec86cae-a4dc-4133-b34d-e19ddc8a244f service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:43.588082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec86cae-a4dc-4133-b34d-e19ddc8a244f service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:43.588356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec86cae-a4dc-4133-b34d-e19ddc8a244f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:43.589958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dec86cae-a4dc-4133-b34d-e19ddc8a244f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:43.680016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec86cae-a4dc-4133-b34d-e19ddc8a244f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:43.681637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec86cae-a4dc-4133-b34d-e19ddc8a244f service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:43.700677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 728/1440] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:45.771031 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423f5b2e-204b-411b-aad4-df65ba6ada30 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:45.771603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423f5b2e-204b-411b-aad4-df65ba6ada30 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:45.771603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423f5b2e-204b-411b-aad4-df65ba6ada30 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:45.772938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 713/1441] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:03:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:45.778913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239a1b2-4b10-4c90-b271-bc775db3a07d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:45.779356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239a1b2-4b10-4c90-b271-bc775db3a07d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:45.779470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f239a1b2-4b10-4c90-b271-bc775db3a07d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:45.819620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:45.846207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-f239a1b2-4b10-4c90-b271-bc775db3a07d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:45.847416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f239a1b2-4b10-4c90-b271-bc775db3a07d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:45.942964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 729/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:03:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:46.229443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423f5b2e-204b-411b-aad4-df65ba6ada30 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:46.231013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-423f5b2e-204b-411b-aad4-df65ba6ada30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:46.336741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423f5b2e-204b-411b-aad4-df65ba6ada30 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:46.370293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:46.412398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7754fd9e-e694-489b-8211-5383e3043c9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:46.413002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7754fd9e-e694-489b-8211-5383e3043c9d service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:46.413279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7754fd9e-e694-489b-8211-5383e3043c9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:46.413552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7754fd9e-e694-489b-8211-5383e3043c9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:46.415168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7754fd9e-e694-489b-8211-5383e3043c9d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:46.523013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7754fd9e-e694-489b-8211-5383e3043c9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:46.524386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7754fd9e-e694-489b-8211-5383e3043c9d service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:46.543169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 714/1443] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:48.847924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad79c77-65b4-40ae-8092-51a9f44f131c None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:48.848632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad79c77-65b4-40ae-8092-51a9f44f131c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:48.848632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad79c77-65b4-40ae-8092-51a9f44f131c None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:48.849834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 730/1444] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:03:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:48.855241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9a93df-7eec-4e6c-b6c3-fb73e35c0e65 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:48.855834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9a93df-7eec-4e6c-b6c3-fb73e35c0e65 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:48.856457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9a93df-7eec-4e6c-b6c3-fb73e35c0e65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:48.922596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-1e9a93df-7eec-4e6c-b6c3-fb73e35c0e65 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:48.923818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1e9a93df-7eec-4e6c-b6c3-fb73e35c0e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:48.936752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:49.020309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 715/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:03:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:49.305681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad79c77-65b4-40ae-8092-51a9f44f131c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:49.307240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9ad79c77-65b4-40ae-8092-51a9f44f131c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:49.411801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad79c77-65b4-40ae-8092-51a9f44f131c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:49.424569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:49.485745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585d8c14-49eb-40c6-b5fa-9fee7acad507 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:49.486347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d8c14-49eb-40c6-b5fa-9fee7acad507 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:49.486625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d8c14-49eb-40c6-b5fa-9fee7acad507 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:49.486892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d8c14-49eb-40c6-b5fa-9fee7acad507 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:49.488507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-585d8c14-49eb-40c6-b5fa-9fee7acad507 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:49.596844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585d8c14-49eb-40c6-b5fa-9fee7acad507 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:49.598289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585d8c14-49eb-40c6-b5fa-9fee7acad507 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:49.616879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 731/1446] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:52.065149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f96389-5b8d-427e-a86a-209dbf7e4aa9 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:52.065714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f96389-5b8d-427e-a86a-209dbf7e4aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:52.065714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f96389-5b8d-427e-a86a-209dbf7e4aa9 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:52.067039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 716/1447] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:03:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:52.072474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e980ad-beca-44a6-935f-6143c5ecf23c None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:52.072899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e980ad-beca-44a6-935f-6143c5ecf23c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:52.073034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e980ad-beca-44a6-935f-6143c5ecf23c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:52.122317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:52.141356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c2e980ad-beca-44a6-935f-6143c5ecf23c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:52.142550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c2e980ad-beca-44a6-935f-6143c5ecf23c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:52.237814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 732/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:03:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:52.528147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f96389-5b8d-427e-a86a-209dbf7e4aa9 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:52.529726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-98f96389-5b8d-427e-a86a-209dbf7e4aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:52.632150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f96389-5b8d-427e-a86a-209dbf7e4aa9 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:52.673077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:52.707422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8315d5cc-daf4-4ad7-95fe-55a6dfd53808 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:52.708055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8315d5cc-daf4-4ad7-95fe-55a6dfd53808 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:52.708332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8315d5cc-daf4-4ad7-95fe-55a6dfd53808 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:52.708602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8315d5cc-daf4-4ad7-95fe-55a6dfd53808 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:52.710186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8315d5cc-daf4-4ad7-95fe-55a6dfd53808 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:52.817488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8315d5cc-daf4-4ad7-95fe-55a6dfd53808 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:52.818905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8315d5cc-daf4-4ad7-95fe-55a6dfd53808 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:52.836744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 717/1449] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:55.145930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d475fb35-5911-4e38-bab4-da1be3e4719e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:55.146455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d475fb35-5911-4e38-bab4-da1be3e4719e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:55.146922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d475fb35-5911-4e38-bab4-da1be3e4719e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:55.149461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 733/1450] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:03:55 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:55.155526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4d5955-ed40-43ad-8667-003f2efd26d3 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:55.155978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4d5955-ed40-43ad-8667-003f2efd26d3 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:55.156153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4d5955-ed40-43ad-8667-003f2efd26d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:55.221551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4a4d5955-ed40-43ad-8667-003f2efd26d3 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:55.222754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4a4d5955-ed40-43ad-8667-003f2efd26d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:55.243559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:55.320018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 718/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:03:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:55.609196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d475fb35-5911-4e38-bab4-da1be3e4719e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:55.610758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d475fb35-5911-4e38-bab4-da1be3e4719e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:55.713073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d475fb35-5911-4e38-bab4-da1be3e4719e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:55.734945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:55.787410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c86be6-ceca-440f-aa78-2e3d3f015848 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:55.788040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c86be6-ceca-440f-aa78-2e3d3f015848 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:55.788320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c86be6-ceca-440f-aa78-2e3d3f015848 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:55.788589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c86be6-ceca-440f-aa78-2e3d3f015848 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:55.790170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d9c86be6-ceca-440f-aa78-2e3d3f015848 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:55.897974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c86be6-ceca-440f-aa78-2e3d3f015848 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:55.899332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c86be6-ceca-440f-aa78-2e3d3f015848 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:55.918310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 734/1452] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:03:58.116690 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3675ab4-d3ed-4fae-8d88-91c4517c3245 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:58.117229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3675ab4-d3ed-4fae-8d88-91c4517c3245 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:58.117229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3675ab4-d3ed-4fae-8d88-91c4517c3245 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:58.118722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 719/1453] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:03:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:03:58.124076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa428110-733a-4d93-954c-8f39e21af2de None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:58.124522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa428110-733a-4d93-954c-8f39e21af2de None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:58.124628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa428110-733a-4d93-954c-8f39e21af2de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:58.188594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:58.191413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-fa428110-733a-4d93-954c-8f39e21af2de None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:03:58.193174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-fa428110-733a-4d93-954c-8f39e21af2de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:58.288920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 735/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:03:58.581980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3675ab4-d3ed-4fae-8d88-91c4517c3245 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:03:58.583510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a3675ab4-d3ed-4fae-8d88-91c4517c3245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:58.686589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3675ab4-d3ed-4fae-8d88-91c4517c3245 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:03:58.736593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:03:58.761444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd10254-0c83-4cc9-9d87-3e0e39c120e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:03:58.762068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd10254-0c83-4cc9-9d87-3e0e39c120e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:03:58.762344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd10254-0c83-4cc9-9d87-3e0e39c120e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:03:58.762614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd10254-0c83-4cc9-9d87-3e0e39c120e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:03:58.764220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5dd10254-0c83-4cc9-9d87-3e0e39c120e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:03:58.871955 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd10254-0c83-4cc9-9d87-3e0e39c120e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:03:58.873307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd10254-0c83-4cc9-9d87-3e0e39c120e4 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:03:58.891888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 720/1455] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:03:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:01.101702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0ebe2-2024-419f-8ae7-32f6c564ba38 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:01.102279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0ebe2-2024-419f-8ae7-32f6c564ba38 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:01.102279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0ebe2-2024-419f-8ae7-32f6c564ba38 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:01.103623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 736/1456] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:04:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:01.109011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036da7bf-c850-44c7-830a-f8fcc42ded09 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:01.109441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036da7bf-c850-44c7-830a-f8fcc42ded09 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:01.109557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036da7bf-c850-44c7-830a-f8fcc42ded09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:01.176806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-036da7bf-c850-44c7-830a-f8fcc42ded09 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:01.178014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-036da7bf-c850-44c7-830a-f8fcc42ded09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:01.209095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:01.277020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 721/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:04:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:01.563547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a0ebe2-2024-419f-8ae7-32f6c564ba38 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:01.565127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-05a0ebe2-2024-419f-8ae7-32f6c564ba38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:01.668914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a0ebe2-2024-419f-8ae7-32f6c564ba38 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:01.697331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:01.744826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0545252a-8a66-4780-85ae-4217f224c9a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:01.745428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545252a-8a66-4780-85ae-4217f224c9a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:01.745709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545252a-8a66-4780-85ae-4217f224c9a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:01.745981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545252a-8a66-4780-85ae-4217f224c9a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:01.747566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0545252a-8a66-4780-85ae-4217f224c9a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:01.856504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0545252a-8a66-4780-85ae-4217f224c9a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:01.857857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0545252a-8a66-4780-85ae-4217f224c9a8 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:01.876508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 737/1458] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:04.168215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e039a1-614a-4c17-a941-0a25b521ff33 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:04.168756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e039a1-614a-4c17-a941-0a25b521ff33 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:04.168885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e039a1-614a-4c17-a941-0a25b521ff33 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:04.170129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 722/1459] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:04:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:04.175433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df13292-dcbc-4ebd-8fd8-dfc93e540377 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:04.175851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df13292-dcbc-4ebd-8fd8-dfc93e540377 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:04.176012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df13292-dcbc-4ebd-8fd8-dfc93e540377 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:04.240805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7df13292-dcbc-4ebd-8fd8-dfc93e540377 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:04.241999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7df13292-dcbc-4ebd-8fd8-dfc93e540377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:04.250343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:04.338172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 738/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:04:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:04.621810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e039a1-614a-4c17-a941-0a25b521ff33 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:04.623347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-26e039a1-614a-4c17-a941-0a25b521ff33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:04.726577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e039a1-614a-4c17-a941-0a25b521ff33 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:04.781694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:04.799668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b222e054-884f-44d9-8daa-0cd56456bf4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:04.800300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b222e054-884f-44d9-8daa-0cd56456bf4e service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:04.800581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b222e054-884f-44d9-8daa-0cd56456bf4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:04.800857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b222e054-884f-44d9-8daa-0cd56456bf4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:04.802438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b222e054-884f-44d9-8daa-0cd56456bf4e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:04.911650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b222e054-884f-44d9-8daa-0cd56456bf4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:04.913010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b222e054-884f-44d9-8daa-0cd56456bf4e service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:04.930979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 723/1461] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:07.151390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da9db39-6607-4ca9-bebc-6d1f74d60e75 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:07.151873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da9db39-6607-4ca9-bebc-6d1f74d60e75 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:07.151987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da9db39-6607-4ca9-bebc-6d1f74d60e75 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:07.153312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 739/1462] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:04:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:07.158523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8834ca92-eef5-4817-ad5a-4dadaa3ac54e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:07.158913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8834ca92-eef5-4817-ad5a-4dadaa3ac54e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:07.159077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8834ca92-eef5-4817-ad5a-4dadaa3ac54e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:07.226259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-8834ca92-eef5-4817-ad5a-4dadaa3ac54e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:07.227519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8834ca92-eef5-4817-ad5a-4dadaa3ac54e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:07.266303 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:07.325111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 724/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:07.608258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da9db39-6607-4ca9-bebc-6d1f74d60e75 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:07.609789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4da9db39-6607-4ca9-bebc-6d1f74d60e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:07.713134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da9db39-6607-4ca9-bebc-6d1f74d60e75 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:07.752232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:07.788121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89504b0-ac0d-43f9-b65e-37299b0bead5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:07.788732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89504b0-ac0d-43f9-b65e-37299b0bead5 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:07.789012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89504b0-ac0d-43f9-b65e-37299b0bead5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:07.789281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89504b0-ac0d-43f9-b65e-37299b0bead5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:07.790903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d89504b0-ac0d-43f9-b65e-37299b0bead5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:07.901256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89504b0-ac0d-43f9-b65e-37299b0bead5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:07.902609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89504b0-ac0d-43f9-b65e-37299b0bead5 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:07.920907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 740/1464] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:10.113157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899e575-abac-4574-a568-eba0104afaa4 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:10.113862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899e575-abac-4574-a568-eba0104afaa4 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:10.113862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899e575-abac-4574-a568-eba0104afaa4 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:10.115054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 725/1465] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:04:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:10.120888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9e884d-4f99-4b18-82cd-09df5bab8ee2 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:10.121333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9e884d-4f99-4b18-82cd-09df5bab8ee2 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:10.121442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9e884d-4f99-4b18-82cd-09df5bab8ee2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:10.186838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-fd9e884d-4f99-4b18-82cd-09df5bab8ee2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:10.188083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-fd9e884d-4f99-4b18-82cd-09df5bab8ee2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:10.207652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:10.285486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 741/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:04:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:10.564908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5899e575-abac-4574-a568-eba0104afaa4 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:10.566467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5899e575-abac-4574-a568-eba0104afaa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:10.670210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5899e575-abac-4574-a568-eba0104afaa4 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:10.736748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:10.743624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f84aba7-ae33-41dd-ac78-b57ffc8327e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:10.744250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f84aba7-ae33-41dd-ac78-b57ffc8327e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:10.744531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f84aba7-ae33-41dd-ac78-b57ffc8327e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:10.744803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f84aba7-ae33-41dd-ac78-b57ffc8327e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:10.746426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4f84aba7-ae33-41dd-ac78-b57ffc8327e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:10.855626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f84aba7-ae33-41dd-ac78-b57ffc8327e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:10.857005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f84aba7-ae33-41dd-ac78-b57ffc8327e8 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:10.875181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 726/1467] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:13.036930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6917f517-5ae9-4fdf-bb0f-6e4d55227d6f None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:13.037530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6917f517-5ae9-4fdf-bb0f-6e4d55227d6f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:13.037530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6917f517-5ae9-4fdf-bb0f-6e4d55227d6f None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:13.038824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 742/1468] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:04:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:13.044848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b86b5-0f72-4d1c-8497-6597b19d1e26 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:13.045258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b86b5-0f72-4d1c-8497-6597b19d1e26 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:13.045395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b86b5-0f72-4d1c-8497-6597b19d1e26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:13.111306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ac5b86b5-0f72-4d1c-8497-6597b19d1e26 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:13.112534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ac5b86b5-0f72-4d1c-8497-6597b19d1e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:13.160856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:13.210833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 727/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:04:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:13.496625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6917f517-5ae9-4fdf-bb0f-6e4d55227d6f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:13.498156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6917f517-5ae9-4fdf-bb0f-6e4d55227d6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:13.601581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6917f517-5ae9-4fdf-bb0f-6e4d55227d6f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:13.649898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:13.676632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c26bc2-3014-46ec-bfe9-813f4f581995 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:13.677234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c26bc2-3014-46ec-bfe9-813f4f581995 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:13.677514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c26bc2-3014-46ec-bfe9-813f4f581995 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:13.677784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c26bc2-3014-46ec-bfe9-813f4f581995 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:13.679379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-80c26bc2-3014-46ec-bfe9-813f4f581995 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:13.788156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c26bc2-3014-46ec-bfe9-813f4f581995 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:13.789597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c26bc2-3014-46ec-bfe9-813f4f581995 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:13.808540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 743/1470] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:23.209335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=22 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:23.805624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=66 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:33.858328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e64387-4785-4a86-b8c2-e63dda55236b None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:33.858904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e64387-4785-4a86-b8c2-e63dda55236b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:33.858904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e64387-4785-4a86-b8c2-e63dda55236b None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:33.860295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 728/1471] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:04:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:33.865666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29cb273-3994-456c-b5a8-94543353224d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:33.866070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29cb273-3994-456c-b5a8-94543353224d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:33.866199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29cb273-3994-456c-b5a8-94543353224d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:33.932227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d29cb273-3994-456c-b5a8-94543353224d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:33.933684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d29cb273-3994-456c-b5a8-94543353224d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:34.029076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 744/1472] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:04:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:34.329841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962762e-dd0d-4ebf-96bc-a4b7227b6bc7 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:34.330124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962762e-dd0d-4ebf-96bc-a4b7227b6bc7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:34.330394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b962762e-dd0d-4ebf-96bc-a4b7227b6bc7 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:34.331757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 729/1473] 10.222.0.27 () {54 vars in 962 bytes} [Thu Sep 15 14:04:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:34.336871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418998cf-8eaf-4a9b-aa8c-337d8f465f02 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:34.337171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418998cf-8eaf-4a9b-aa8c-337d8f465f02 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:34.337456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418998cf-8eaf-4a9b-aa8c-337d8f465f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:34.414241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-418998cf-8eaf-4a9b-aa8c-337d8f465f02 None None] MFA Rules not processed for user `54db2e6805fd40f79c03eb3b2319f2b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:34.415816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-418998cf-8eaf-4a9b-aa8c-337d8f465f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:34.436785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:34.492194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 745/1474] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Sep 15 14:04:34 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:34.500505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b962762e-dd0d-4ebf-96bc-a4b7227b6bc7 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:34.502017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b962762e-dd0d-4ebf-96bc-a4b7227b6bc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:34.603879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b962762e-dd0d-4ebf-96bc-a4b7227b6bc7 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:34.675977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8443a2-1677-49ab-9bf1-6d9a481710d1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '54db2e6805fd40f79c03eb3b2319f2b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:34.676576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8443a2-1677-49ab-9bf1-6d9a481710d1 service glance] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:34.676856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8443a2-1677-49ab-9bf1-6d9a481710d1 service glance] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:34.677130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8443a2-1677-49ab-9bf1-6d9a481710d1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:34.678715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ca8443a2-1677-49ab-9bf1-6d9a481710d1 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:34.732774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:34.788979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8443a2-1677-49ab-9bf1-6d9a481710d1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:34.790920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8443a2-1677-49ab-9bf1-6d9a481710d1 service glance] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:34.809748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 730/1475] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:37.043386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7fe901-99fd-435a-b877-f0f9cb8084e0 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:37.044013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7fe901-99fd-435a-b877-f0f9cb8084e0 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:37.044013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7fe901-99fd-435a-b877-f0f9cb8084e0 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:37.045305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 746/1476] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:04:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:37.050781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbde42c-4071-4999-876b-cce57a2e369f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:37.051178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbde42c-4071-4999-876b-cce57a2e369f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:37.051317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbde42c-4071-4999-876b-cce57a2e369f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:37.117285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-dcbde42c-4071-4999-876b-cce57a2e369f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:37.118487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dcbde42c-4071-4999-876b-cce57a2e369f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:37.213692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 731/1477] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:04:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:37.489549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7fe901-99fd-435a-b877-f0f9cb8084e0 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:37.491096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9c7fe901-99fd-435a-b877-f0f9cb8084e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:37.617565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7fe901-99fd-435a-b877-f0f9cb8084e0 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:37.667253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:37.712760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e281584-c693-4dae-9c13-4867d57f57db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:37.713353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e281584-c693-4dae-9c13-4867d57f57db admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:37.713628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e281584-c693-4dae-9c13-4867d57f57db admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:37.713896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e281584-c693-4dae-9c13-4867d57f57db admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:37.714530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e281584-c693-4dae-9c13-4867d57f57db admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:37.716316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING py.warnings [None req-5e281584-c693-4dae-9c13-4867d57f57db admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:04:37.716316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: warnings.warn(msg) Sep 15 14:04:37.716316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: Sep 15 14:04:37.717474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e281584-c693-4dae-9c13-4867d57f57db admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:37.736933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 747/1478] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 15 14:04:37 2022] GET /identity/v3/projects => generated 2553 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:04:39.912362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcad6138-89ee-41a6-8ee3-d635f450ff7f None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:39.912855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcad6138-89ee-41a6-8ee3-d635f450ff7f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:39.912989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcad6138-89ee-41a6-8ee3-d635f450ff7f None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:39.914464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 732/1479] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:04:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:39.919860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bd0de0-a289-42e3-a03d-586df9b9264b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:39.920276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bd0de0-a289-42e3-a03d-586df9b9264b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:39.920410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bd0de0-a289-42e3-a03d-586df9b9264b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:39.987747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e8bd0de0-a289-42e3-a03d-586df9b9264b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:39.988929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e8bd0de0-a289-42e3-a03d-586df9b9264b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:40.084752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 748/1480] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:04:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:40.384798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcad6138-89ee-41a6-8ee3-d635f450ff7f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:40.386354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bcad6138-89ee-41a6-8ee3-d635f450ff7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:40.396365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:40.530260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcad6138-89ee-41a6-8ee3-d635f450ff7f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:40.609127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:40.641751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b889110a-c89f-4455-9efa-cb4690adcef1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:40.642367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889110a-c89f-4455-9efa-cb4690adcef1 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:40.642652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889110a-c89f-4455-9efa-cb4690adcef1 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:40.642929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889110a-c89f-4455-9efa-cb4690adcef1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:40.644582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b889110a-c89f-4455-9efa-cb4690adcef1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:40.734769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b889110a-c89f-4455-9efa-cb4690adcef1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:40.736373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b889110a-c89f-4455-9efa-cb4690adcef1 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:40.754200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 733/1481] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:43.082708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29acb5c-143a-4ab8-b1c2-cc1200c676c3 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:43.083294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29acb5c-143a-4ab8-b1c2-cc1200c676c3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:43.083294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29acb5c-143a-4ab8-b1c2-cc1200c676c3 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:43.084626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 749/1482] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:04:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:43.090609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a48204-dbe6-42f4-b0d5-4fe8e7dff971 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:43.091022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a48204-dbe6-42f4-b0d5-4fe8e7dff971 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:43.091160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a48204-dbe6-42f4-b0d5-4fe8e7dff971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:43.156678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-44a48204-dbe6-42f4-b0d5-4fe8e7dff971 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:43.157886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-44a48204-dbe6-42f4-b0d5-4fe8e7dff971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:43.205176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:43.255189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 734/1483] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:04:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:43.545991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29acb5c-143a-4ab8-b1c2-cc1200c676c3 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:43.547525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c29acb5c-143a-4ab8-b1c2-cc1200c676c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:43.572267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:43.654959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29acb5c-143a-4ab8-b1c2-cc1200c676c3 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:43.751773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4793e4b3-2517-42ba-a06a-dc81bddbd823 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:43.752381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4793e4b3-2517-42ba-a06a-dc81bddbd823 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:43.752657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4793e4b3-2517-42ba-a06a-dc81bddbd823 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:43.752929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4793e4b3-2517-42ba-a06a-dc81bddbd823 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:43.754957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4793e4b3-2517-42ba-a06a-dc81bddbd823 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:43.841918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:43.883839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4793e4b3-2517-42ba-a06a-dc81bddbd823 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:43.885306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4793e4b3-2517-42ba-a06a-dc81bddbd823 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:43.910652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 750/1484] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:46.162847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8bef8f-a6c5-4556-ac62-24046d7e34dc None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:46.163424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8bef8f-a6c5-4556-ac62-24046d7e34dc None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:46.163424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8bef8f-a6c5-4556-ac62-24046d7e34dc None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:46.165001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 735/1485] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:04:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:46.170212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab02139-7035-41b5-bb83-25b2d3f2fa14 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:46.170643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab02139-7035-41b5-bb83-25b2d3f2fa14 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:46.170777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab02139-7035-41b5-bb83-25b2d3f2fa14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:46.236946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-dab02139-7035-41b5-bb83-25b2d3f2fa14 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:46.238166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dab02139-7035-41b5-bb83-25b2d3f2fa14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:46.335231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 751/1486] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:04:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:46.624957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8bef8f-a6c5-4556-ac62-24046d7e34dc None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:46.626478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3b8bef8f-a6c5-4556-ac62-24046d7e34dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:46.731301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8bef8f-a6c5-4556-ac62-24046d7e34dc None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:46.802128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:46.803617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34381d3c-f7ed-4d1c-89e8-b0661d9f1537 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:46.805180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34381d3c-f7ed-4d1c-89e8-b0661d9f1537 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:46.805749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34381d3c-f7ed-4d1c-89e8-b0661d9f1537 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:46.806199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34381d3c-f7ed-4d1c-89e8-b0661d9f1537 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:46.808357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-34381d3c-f7ed-4d1c-89e8-b0661d9f1537 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:46.917608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34381d3c-f7ed-4d1c-89e8-b0661d9f1537 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:46.919027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34381d3c-f7ed-4d1c-89e8-b0661d9f1537 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:46.937469 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 736/1487] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:49.198592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85165ff-4752-4f83-8850-c62a3844cf5e None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:49.199123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85165ff-4752-4f83-8850-c62a3844cf5e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:49.199233 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85165ff-4752-4f83-8850-c62a3844cf5e None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:49.200593 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 752/1488] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:04:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:49.205927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d548400-c215-4231-95b4-2aa2c57de815 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:49.206396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d548400-c215-4231-95b4-2aa2c57de815 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:49.206504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d548400-c215-4231-95b4-2aa2c57de815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:49.274460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6d548400-c215-4231-95b4-2aa2c57de815 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:49.275647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6d548400-c215-4231-95b4-2aa2c57de815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:49.332939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:49.379111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 737/1489] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:04:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:49.681841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85165ff-4752-4f83-8850-c62a3844cf5e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:49.683653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d85165ff-4752-4f83-8850-c62a3844cf5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:49.719635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:49.786328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85165ff-4752-4f83-8850-c62a3844cf5e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:49.860668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b825382-9e90-42fe-b082-7bc397f96341 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:49.861266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b825382-9e90-42fe-b082-7bc397f96341 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:49.861551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b825382-9e90-42fe-b082-7bc397f96341 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:49.861818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b825382-9e90-42fe-b082-7bc397f96341 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:49.863394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3b825382-9e90-42fe-b082-7bc397f96341 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:49.943406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:49.973533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b825382-9e90-42fe-b082-7bc397f96341 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:49.974970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b825382-9e90-42fe-b082-7bc397f96341 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:49.993722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 753/1490] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:04:52.324328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d793b08-7b7e-468c-977e-77af1e875e00 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:52.324746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d793b08-7b7e-468c-977e-77af1e875e00 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:52.324862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d793b08-7b7e-468c-977e-77af1e875e00 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:52.326219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 738/1491] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 15 14:04:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:04:52.331597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e77623-f67e-4c45-9c3d-209a729f13e1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:52.332030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e77623-f67e-4c45-9c3d-209a729f13e1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:52.332196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e77623-f67e-4c45-9c3d-209a729f13e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:52.400150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-82e77623-f67e-4c45-9c3d-209a729f13e1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:04:52.401355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-82e77623-f67e-4c45-9c3d-209a729f13e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:52.496893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 754/1492] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 15 14:04:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:04:52.791248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d793b08-7b7e-468c-977e-77af1e875e00 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:04:52.792815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7d793b08-7b7e-468c-977e-77af1e875e00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:52.895499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d793b08-7b7e-468c-977e-77af1e875e00 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:04:52.969139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fef0f49-95c7-4cda-b75f-00ddbfd4a23e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:04:52.969740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef0f49-95c7-4cda-b75f-00ddbfd4a23e service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:04:52.970019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef0f49-95c7-4cda-b75f-00ddbfd4a23e service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:04:52.970288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef0f49-95c7-4cda-b75f-00ddbfd4a23e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:04:52.971899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8fef0f49-95c7-4cda-b75f-00ddbfd4a23e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:04:52.981127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:04:53.079284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fef0f49-95c7-4cda-b75f-00ddbfd4a23e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:04:53.080689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fef0f49-95c7-4cda-b75f-00ddbfd4a23e service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:04:53.098895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 739/1493] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:04:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:05:02.495333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=82 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:05:03.096719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=50 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:09.267788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28e585-c31f-46ff-9a04-13795e055be6 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:09.268352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28e585-c31f-46ff-9a04-13795e055be6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:09.268352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28e585-c31f-46ff-9a04-13795e055be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:09.349146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8e28e585-c31f-46ff-9a04-13795e055be6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:06:09.350383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8e28e585-c31f-46ff-9a04-13795e055be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:09.444513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 755/1494] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:06:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:06:09.452501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674184bf-c088-4bc2-8a41-eafa58690617 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:09.453008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674184bf-c088-4bc2-8a41-eafa58690617 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:09.453008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674184bf-c088-4bc2-8a41-eafa58690617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:09.530206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-674184bf-c088-4bc2-8a41-eafa58690617 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:06:09.531402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-674184bf-c088-4bc2-8a41-eafa58690617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:09.626902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 740/1495] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:06:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:06:09.631545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e28e585-c31f-46ff-9a04-13795e055be6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:09.632781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8e28e585-c31f-46ff-9a04-13795e055be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:09.644754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e28e585-c31f-46ff-9a04-13795e055be6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:09.705575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:09.741150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:09.741785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:09.742063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:09.742333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:09.742972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:06:09.743274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:09.744308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING py.warnings [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:06:09.744308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: warnings.warn(msg) Sep 15 14:06:09.744308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: Sep 15 14:06:09.745201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:09.756703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 756/1496] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:06:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:06:09.765007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674184bf-c088-4bc2-8a41-eafa58690617 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:09.774994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674184bf-c088-4bc2-8a41-eafa58690617 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:09.840764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:09.874503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:09.875111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:09.875382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:09.875645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:09.876292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:09.878776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING py.warnings [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:06:09.878776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: warnings.warn(msg) Sep 15 14:06:09.878776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: Sep 15 14:06:09.880068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:09.935606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 741/1497] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:06:09 2022] POST /identity/v3/projects => generated 379 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:06:09.939857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:09.941908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:09.976530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5156f51-d28d-4326-bc75-f4d1ba081844 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:10.070071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:10.070677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:10.070957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:10.071235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:10.071885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:10.072890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING py.warnings [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:06:10.072890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: warnings.warn(msg) Sep 15 14:06:10.072890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: Sep 15 14:06:10.074121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:10.116360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 757/1498] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:06:09 2022] POST /identity/v3/users => generated 290 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:06:10.120564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:10.122122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:10.183942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e153c-e5c6-4332-ac42-66c174108c6a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:10.269459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:10.280611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:10.281209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:10.281482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:10.281750 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:10.282380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:10.283371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING py.warnings [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:06:10.283371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: warnings.warn(msg) Sep 15 14:06:10.283371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: Sep 15 14:06:10.284361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:10.296393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 742/1499] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:06:10 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:06:10.301167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:10.302693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:10.314659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b8951-2d42-4ffe-a78f-5113e6e3ebac admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:10.339706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:10.408836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3f4e8a-23d9-4cb2-8880-23d3a477866c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:10.409444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3f4e8a-23d9-4cb2-8880-23d3a477866c admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:10.409723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3f4e8a-23d9-4cb2-8880-23d3a477866c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:10.409996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3f4e8a-23d9-4cb2-8880-23d3a477866c admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:10.410637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3f4e8a-23d9-4cb2-8880-23d3a477866c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:10.411642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING py.warnings [None req-2b3f4e8a-23d9-4cb2-8880-23d3a477866c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:06:10.411642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: warnings.warn(msg) Sep 15 14:06:10.411642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: Sep 15 14:06:10.412555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3f4e8a-23d9-4cb2-8880-23d3a477866c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:10.425178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 758/1500] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:06:10 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:06:10.429673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:10.431464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:10.466147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6feb94f-d5d0-4dc7-8c06-122f7dd08a75 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:10.560069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5c819f-67fd-425a-80a2-f36ae32a0903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:10.560638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5c819f-67fd-425a-80a2-f36ae32a0903 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:10.560921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5c819f-67fd-425a-80a2-f36ae32a0903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:10.561193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5c819f-67fd-425a-80a2-f36ae32a0903 admin admin] PATH_INFO: `/v3/projects/a5ebf308f3824009a115f990b7938353/users/91fdd047ba29496cb0e0b7f06ba55426/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:10.583200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5c819f-67fd-425a-80a2-f36ae32a0903 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5ebf308f3824009a115f990b7938353, user_id=91fdd047ba29496cb0e0b7f06ba55426, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:10.584316 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING py.warnings [None req-2a5c819f-67fd-425a-80a2-f36ae32a0903 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:06:10.584316 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: warnings.warn(msg) Sep 15 14:06:10.584316 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: Sep 15 14:06:10.586269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5c819f-67fd-425a-80a2-f36ae32a0903 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:10.610157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 743/1501] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:06:10 2022] PUT /identity/v3/projects/a5ebf308f3824009a115f990b7938353/users/91fdd047ba29496cb0e0b7f06ba55426/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:06:10.623309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a5f43-d614-4baa-b85b-c5eba8621b33 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:10.623599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a5f43-d614-4baa-b85b-c5eba8621b33 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:10.623918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a5f43-d614-4baa-b85b-c5eba8621b33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:10.681028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9a9a5f43-d614-4baa-b85b-c5eba8621b33 None None] MFA Rules not processed for user `91fdd047ba29496cb0e0b7f06ba55426`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:06:10.682486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9a9a5f43-d614-4baa-b85b-c5eba8621b33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:10.758979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 759/1502] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:06:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:06:10.772260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452d584b-d100-4061-9dac-ed9d54798dbc None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:10.772550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452d584b-d100-4061-9dac-ed9d54798dbc None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:10.772828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452d584b-d100-4061-9dac-ed9d54798dbc None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:10.774164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 744/1503] 10.222.0.27 () {54 vars in 964 bytes} [Thu Sep 15 14:06:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:06:10.779027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79def636-e9ad-4cba-a88f-5de323fa03af None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:10.779341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79def636-e9ad-4cba-a88f-5de323fa03af None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:10.779613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79def636-e9ad-4cba-a88f-5de323fa03af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:10.811805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:10.973324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-79def636-e9ad-4cba-a88f-5de323fa03af None None] MFA Rules not processed for user `f54979e322484a42bb57d1b3a619652c`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:06:10.974537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-79def636-e9ad-4cba-a88f-5de323fa03af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:11.049322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 760/1504] 10.222.0.27 () {58 vars in 1061 bytes} [Thu Sep 15 14:06:10 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:06:11.059403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452d584b-d100-4061-9dac-ed9d54798dbc None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:11.060971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-452d584b-d100-4061-9dac-ed9d54798dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:11.111418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:11.165038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452d584b-d100-4061-9dac-ed9d54798dbc None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:11.239516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:11.240141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:11.240421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:11.240692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:11.242282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:11.341238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:11.342698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:11.346140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:11.360896 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 745/1505] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:06:11 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:06:11.406414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15625e9-44b3-438b-8559-32369711dbad None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:11.406808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15625e9-44b3-438b-8559-32369711dbad None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:11.406994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15625e9-44b3-438b-8559-32369711dbad None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:11.408267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 761/1506] 10.222.0.27 () {52 vars in 830 bytes} [Thu Sep 15 14:06:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:06:11.412334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:11.413555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:11.513038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35036fc-46b7-4f30-9782-61e3fb5fa1e4 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:11.605627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:11.606254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:11.606541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:11.606815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] PATH_INFO: `/v3/users/91fdd047ba29496cb0e0b7f06ba55426` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:11.620819 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] RBAC: Authorizing `identity:delete_user(user_id=91fdd047ba29496cb0e0b7f06ba55426)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:11.621764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING py.warnings [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:06:11.621764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: warnings.warn(msg) Sep 15 14:06:11.621764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: Sep 15 14:06:11.622993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:11.633403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:11.689430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91fdd047ba29496cb0e0b7f06ba55426'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:11.699320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91fdd047ba29496cb0e0b7f06ba55426'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:11.699803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91fdd047ba29496cb0e0b7f06ba55426'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:11.700341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91fdd047ba29496cb0e0b7f06ba55426'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:11.714214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91fdd047ba29496cb0e0b7f06ba55426'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:11.744390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 746/1507] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:06:11 2022] DELETE /identity/v3/users/91fdd047ba29496cb0e0b7f06ba55426 => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:06:11.748864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa943d7-9fc6-487a-a480-9a19ecd6009e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:11.749157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa943d7-9fc6-487a-a480-9a19ecd6009e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:11.749429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa943d7-9fc6-487a-a480-9a19ecd6009e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:11.806723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-faa943d7-9fc6-487a-a480-9a19ecd6009e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:06:11.807937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-faa943d7-9fc6-487a-a480-9a19ecd6009e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:11.883119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:11.894984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 762/1508] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:06:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:06:11.904648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:11.905868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:12.006661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f95880-d084-4987-a7af-e70595dbad3c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:12.063695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:12.084297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e26a089-526a-4282-8e42-c763e09be971 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:12.084910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26a089-526a-4282-8e42-c763e09be971 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:12.085190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26a089-526a-4282-8e42-c763e09be971 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:12.085470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26a089-526a-4282-8e42-c763e09be971 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:12.087064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5e26a089-526a-4282-8e42-c763e09be971 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:12.195050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e26a089-526a-4282-8e42-c763e09be971 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:12.196427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e26a089-526a-4282-8e42-c763e09be971 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:12.214636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 747/1509] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:06:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:06:12.574675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa943d7-9fc6-487a-a480-9a19ecd6009e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:06:12.575934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-faa943d7-9fc6-487a-a480-9a19ecd6009e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:06:12.587901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa943d7-9fc6-487a-a480-9a19ecd6009e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:06:12.686981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:06:12.687614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:06:12.687926 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:06:12.688199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] PATH_INFO: `/v3/projects/a5ebf308f3824009a115f990b7938353` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:06:12.699164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5ebf308f3824009a115f990b7938353)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:06:12.700166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING py.warnings [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:06:12.700166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: warnings.warn(msg) Sep 15 14:06:12.700166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: Sep 15 14:06:12.701362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:06:12.759496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5ebf308f3824009a115f990b7938353', 'request_id': 'req-ebfb5354-728d-42e4-8beb-3cb5842b7476'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:12.770821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5ebf308f3824009a115f990b7938353', 'request_id': 'req-ebfb5354-728d-42e4-8beb-3cb5842b7476'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:12.777015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5ebf308f3824009a115f990b7938353', 'request_id': 'req-ebfb5354-728d-42e4-8beb-3cb5842b7476'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:12.806682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] The token cache is being invalidate because project a5ebf308f3824009a115f990b7938353 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:06:12.806974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:12.807400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ebfb5354-728d-42e4-8beb-3cb5842b7476 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:06:12.808813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 763/1510] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:06:12 2022] DELETE /identity/v3/projects/a5ebf308f3824009a115f990b7938353 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:06:22.212373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=62 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:06:22.804416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:07:39.709243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6988f-eab3-4927-a7c0-7a2014c0d0aa None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:39.709844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6988f-eab3-4927-a7c0-7a2014c0d0aa None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:39.709844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6988f-eab3-4927-a7c0-7a2014c0d0aa None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:39.711135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 748/1511] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:07:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:07:39.716117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c686856f-27c4-4334-b742-0a9d0dea9cc5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:39.716601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c686856f-27c4-4334-b742-0a9d0dea9cc5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:39.716709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c686856f-27c4-4334-b742-0a9d0dea9cc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:39.796207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c686856f-27c4-4334-b742-0a9d0dea9cc5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:07:39.797408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c686856f-27c4-4334-b742-0a9d0dea9cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:39.898719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 764/1512] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:07:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:07:40.060807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c6988f-eab3-4927-a7c0-7a2014c0d0aa None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:07:40.062912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-32c6988f-eab3-4927-a7c0-7a2014c0d0aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:40.181663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c6988f-eab3-4927-a7c0-7a2014c0d0aa None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:07:40.187189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0778d206-3cf8-4ce4-9a84-3d5c4ff4ade4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:07:40.187831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778d206-3cf8-4ce4-9a84-3d5c4ff4ade4 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:40.188114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778d206-3cf8-4ce4-9a84-3d5c4ff4ade4 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:40.188396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778d206-3cf8-4ce4-9a84-3d5c4ff4ade4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:40.197000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0778d206-3cf8-4ce4-9a84-3d5c4ff4ade4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:07:40.198652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0778d206-3cf8-4ce4-9a84-3d5c4ff4ade4 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:07:40.208768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 749/1513] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:07:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:07:41.294254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc8e1e-3549-4d2b-b22a-1c2718378691 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:41.294732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc8e1e-3549-4d2b-b22a-1c2718378691 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:41.294846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bc8e1e-3549-4d2b-b22a-1c2718378691 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:41.296195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 765/1514] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:07:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:07:41.302877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ea33ae-b29d-4237-a60f-df34040c9ccb None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:41.303329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ea33ae-b29d-4237-a60f-df34040c9ccb None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:41.303504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ea33ae-b29d-4237-a60f-df34040c9ccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:41.324149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-64ea33ae-b29d-4237-a60f-df34040c9ccb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:07:41.325340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-64ea33ae-b29d-4237-a60f-df34040c9ccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:41.333813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 750/1515] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:07:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:07:41.483883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bc8e1e-3549-4d2b-b22a-1c2718378691 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:07:41.498351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bc8e1e-3549-4d2b-b22a-1c2718378691 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:07:41.504202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86837b-6a4b-4316-b344-8d22dce8483e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:07:41.504789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86837b-6a4b-4316-b344-8d22dce8483e service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:41.505065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86837b-6a4b-4316-b344-8d22dce8483e service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:41.505334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86837b-6a4b-4316-b344-8d22dce8483e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:41.513290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb86837b-6a4b-4316-b344-8d22dce8483e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:07:41.514844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb86837b-6a4b-4316-b344-8d22dce8483e service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:07:41.525010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 766/1516] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:07:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:07:41.806915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106212a-e664-4fbe-9074-9486af19715f None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:41.807137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106212a-e664-4fbe-9074-9486af19715f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:41.807387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106212a-e664-4fbe-9074-9486af19715f None None] PATH_INFO: `/v3` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:41.808706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 751/1517] 10.222.0.52 () {54 vars in 925 bytes} [Thu Sep 15 14:07:41 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:07:41.815718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1a8e5b-cb87-4ec3-98d6-04bf3ad26681 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:41.816011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1a8e5b-cb87-4ec3-98d6-04bf3ad26681 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:41.816331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1a8e5b-cb87-4ec3-98d6-04bf3ad26681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:41.874381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ac1a8e5b-cb87-4ec3-98d6-04bf3ad26681 None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:07:41.875575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ac1a8e5b-cb87-4ec3-98d6-04bf3ad26681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:41.892930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:07:41.947523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 767/1518] 10.222.0.52 () {58 vars in 1015 bytes} [Thu Sep 15 14:07:41 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:07:50.733657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e74c96-b84c-41d8-8417-f43ed4d8e713 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:50.734113 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e74c96-b84c-41d8-8417-f43ed4d8e713 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:50.734286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e74c96-b84c-41d8-8417-f43ed4d8e713 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:50.735619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 752/1519] 10.222.0.27 () {54 vars in 961 bytes} [Thu Sep 15 14:07:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:07:50.740834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb312a-6540-45a7-91b4-c0e41fb4d322 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:50.741284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb312a-6540-45a7-91b4-c0e41fb4d322 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:50.741383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb312a-6540-45a7-91b4-c0e41fb4d322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:50.760645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-18bb312a-6540-45a7-91b4-c0e41fb4d322 None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:07:50.761819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-18bb312a-6540-45a7-91b4-c0e41fb4d322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:50.770860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 768/1520] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Sep 15 14:07:50 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:07:50.778408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e74c96-b84c-41d8-8417-f43ed4d8e713 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:07:50.792202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e74c96-b84c-41d8-8417-f43ed4d8e713 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:07:50.797136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241035fd-1f0c-4243-b6b7-192e906b806c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:07:50.797756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241035fd-1f0c-4243-b6b7-192e906b806c service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:50.798188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241035fd-1f0c-4243-b6b7-192e906b806c service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:50.798466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241035fd-1f0c-4243-b6b7-192e906b806c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:50.806805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241035fd-1f0c-4243-b6b7-192e906b806c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:07:50.808416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241035fd-1f0c-4243-b6b7-192e906b806c service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:07:50.817872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 753/1521] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:07:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:07:53.002137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5084bb7c-e40c-4f84-ace5-23804010f752 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:53.002711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5084bb7c-e40c-4f84-ace5-23804010f752 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:53.002711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5084bb7c-e40c-4f84-ace5-23804010f752 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:53.004048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 769/1522] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:07:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:07:53.010241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c46f8b-e9e7-47da-ae67-e364a5a94782 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:53.010664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c46f8b-e9e7-47da-ae67-e364a5a94782 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:53.010794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c46f8b-e9e7-47da-ae67-e364a5a94782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:53.029817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-44c46f8b-e9e7-47da-ae67-e364a5a94782 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:07:53.031005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-44c46f8b-e9e7-47da-ae67-e364a5a94782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:53.038764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 754/1523] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:07:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:07:53.194085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5084bb7c-e40c-4f84-ace5-23804010f752 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:07:53.208686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5084bb7c-e40c-4f84-ace5-23804010f752 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:07:53.214045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beffb381-e26c-449b-968f-f7fddc5cd417 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:07:53.214626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beffb381-e26c-449b-968f-f7fddc5cd417 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:53.214903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beffb381-e26c-449b-968f-f7fddc5cd417 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:53.215174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beffb381-e26c-449b-968f-f7fddc5cd417 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:53.223935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beffb381-e26c-449b-968f-f7fddc5cd417 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:07:53.225509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beffb381-e26c-449b-968f-f7fddc5cd417 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:07:53.234675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 770/1524] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:07:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:07:54.307605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40d607-f120-4327-924f-3554a0479e88 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:54.308192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40d607-f120-4327-924f-3554a0479e88 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:54.308192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40d607-f120-4327-924f-3554a0479e88 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:54.309522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 755/1525] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 15 14:07:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:07:54.316691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea74a98d-4d5a-489d-8861-0f58f39627fe None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:54.317115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea74a98d-4d5a-489d-8861-0f58f39627fe None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:54.317257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea74a98d-4d5a-489d-8861-0f58f39627fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:54.336450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ea74a98d-4d5a-489d-8861-0f58f39627fe None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:07:54.337650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ea74a98d-4d5a-489d-8861-0f58f39627fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:54.345427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 771/1526] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 15 14:07:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:07:54.497000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce40d607-f120-4327-924f-3554a0479e88 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:07:54.511808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce40d607-f120-4327-924f-3554a0479e88 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:07:54.517499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc3b7d-ca34-426d-8d65-d002776943ff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:07:54.518081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc3b7d-ca34-426d-8d65-d002776943ff service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:54.518358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc3b7d-ca34-426d-8d65-d002776943ff service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:54.518626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc3b7d-ca34-426d-8d65-d002776943ff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:54.526593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fc3b7d-ca34-426d-8d65-d002776943ff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:07:54.528185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fc3b7d-ca34-426d-8d65-d002776943ff service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:07:54.537597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 756/1527] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:07:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:07:58.469006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb6c01f-bf98-4d80-95d0-5ca2f14fb33a None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:58.469589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb6c01f-bf98-4d80-95d0-5ca2f14fb33a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:58.469698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb6c01f-bf98-4d80-95d0-5ca2f14fb33a None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:58.471039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 772/1528] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:07:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:07:58.476147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d03d92-d476-4b67-b2e9-9e73ad6abb88 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:58.476580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d03d92-d476-4b67-b2e9-9e73ad6abb88 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:58.476683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d03d92-d476-4b67-b2e9-9e73ad6abb88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:58.496273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-a0d03d92-d476-4b67-b2e9-9e73ad6abb88 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:07:58.497468 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a0d03d92-d476-4b67-b2e9-9e73ad6abb88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:58.506036 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 757/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:07:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:07:58.792293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb6c01f-bf98-4d80-95d0-5ca2f14fb33a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:07:58.794013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-beb6c01f-bf98-4d80-95d0-5ca2f14fb33a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:07:58.872934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb6c01f-bf98-4d80-95d0-5ca2f14fb33a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:07:58.877334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90062ba5-ac6c-4251-bf4f-1eab4479edd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:07:58.877941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90062ba5-ac6c-4251-bf4f-1eab4479edd0 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:07:58.878220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90062ba5-ac6c-4251-bf4f-1eab4479edd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:07:58.878490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90062ba5-ac6c-4251-bf4f-1eab4479edd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:07:58.888454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90062ba5-ac6c-4251-bf4f-1eab4479edd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:07:58.889899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90062ba5-ac6c-4251-bf4f-1eab4479edd0 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:07:58.899457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 773/1530] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:07:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:08:01.061573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06da4546-225e-447b-9242-09b24926091e None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:01.062155 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06da4546-225e-447b-9242-09b24926091e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:01.062155 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06da4546-225e-447b-9242-09b24926091e None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:01.064447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 758/1531] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:08:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:08:01.069664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2336c87-8eee-43c6-8da9-02dda87d088c None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:01.070119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2336c87-8eee-43c6-8da9-02dda87d088c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:01.070237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2336c87-8eee-43c6-8da9-02dda87d088c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:01.089582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a2336c87-8eee-43c6-8da9-02dda87d088c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:08:01.090759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a2336c87-8eee-43c6-8da9-02dda87d088c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:08:01.098707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 774/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:08:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:08:01.388186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06da4546-225e-447b-9242-09b24926091e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:08:01.398820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06da4546-225e-447b-9242-09b24926091e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:08:01.402242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d88895-be05-4c15-a029-58d9999e349e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:08:01.402833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d88895-be05-4c15-a029-58d9999e349e service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:01.403109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d88895-be05-4c15-a029-58d9999e349e service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:01.403379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d88895-be05-4c15-a029-58d9999e349e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:01.412824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d88895-be05-4c15-a029-58d9999e349e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:08:01.414259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d88895-be05-4c15-a029-58d9999e349e service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:08:01.424464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 759/1533] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:08:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:08:05.082687 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc86668-bbe8-4e9b-8e58-1d749556bf2c None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:05.083249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc86668-bbe8-4e9b-8e58-1d749556bf2c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:05.083249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc86668-bbe8-4e9b-8e58-1d749556bf2c None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:05.084618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 775/1534] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:08:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:08:05.089867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bfb4c-83cc-4c00-bb81-67342731f96c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:05.090305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bfb4c-83cc-4c00-bb81-67342731f96c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:05.090411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800bfb4c-83cc-4c00-bb81-67342731f96c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:05.098732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=81 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:08:05.111397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-800bfb4c-83cc-4c00-bb81-67342731f96c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:08:05.112701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-800bfb4c-83cc-4c00-bb81-67342731f96c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:08:05.120496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 760/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:08:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:08:05.422008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc86668-bbe8-4e9b-8e58-1d749556bf2c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:08:05.423755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7dc86668-bbe8-4e9b-8e58-1d749556bf2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:08:05.489095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:08:05.503566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc86668-bbe8-4e9b-8e58-1d749556bf2c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:08:05.507204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cefc3a-097c-4fb9-b7e3-c18a8fb779a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:08:05.507841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cefc3a-097c-4fb9-b7e3-c18a8fb779a3 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:05.508125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cefc3a-097c-4fb9-b7e3-c18a8fb779a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:05.508402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cefc3a-097c-4fb9-b7e3-c18a8fb779a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:05.518074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cefc3a-097c-4fb9-b7e3-c18a8fb779a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:08:05.519538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cefc3a-097c-4fb9-b7e3-c18a8fb779a3 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:08:05.528951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 776/1536] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:08:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:08:07.681198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bb2f0d-8c0e-4feb-b701-c8dbfbbcde52 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:07.681726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bb2f0d-8c0e-4feb-b701-c8dbfbbcde52 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:07.681726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bb2f0d-8c0e-4feb-b701-c8dbfbbcde52 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:07.683067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 761/1537] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 15 14:08:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:08:07.688452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0722a2-6c3e-4a3d-a1c8-82b5ac9a77cf None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:07.688879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0722a2-6c3e-4a3d-a1c8-82b5ac9a77cf None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:07.689032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0722a2-6c3e-4a3d-a1c8-82b5ac9a77cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:07.708404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6e0722a2-6c3e-4a3d-a1c8-82b5ac9a77cf None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:08:07.709590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6e0722a2-6c3e-4a3d-a1c8-82b5ac9a77cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:08:07.717336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 777/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 15 14:08:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:08:08.023311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bb2f0d-8c0e-4feb-b701-c8dbfbbcde52 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:08:08.025010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-95bb2f0d-8c0e-4feb-b701-c8dbfbbcde52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:08:08.038650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bb2f0d-8c0e-4feb-b701-c8dbfbbcde52 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:08:08.042405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cb951c-5a84-4504-aa4d-3714f1670738 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:08:08.043033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb951c-5a84-4504-aa4d-3714f1670738 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:08:08.043311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb951c-5a84-4504-aa4d-3714f1670738 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:08:08.043582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb951c-5a84-4504-aa4d-3714f1670738 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:08:08.052949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cb951c-5a84-4504-aa4d-3714f1670738 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:08:08.054450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cb951c-5a84-4504-aa4d-3714f1670738 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:08:08.064563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 762/1539] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:08:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:08:17.703453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=11 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:08:18.059898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=8 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:32.581041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1320dd-2bad-41cd-9431-81212c296f04 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.581669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1320dd-2bad-41cd-9431-81212c296f04 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.581669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1320dd-2bad-41cd-9431-81212c296f04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.613582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-4c1320dd-2bad-41cd-9431-81212c296f04 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:32.614789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4c1320dd-2bad-41cd-9431-81212c296f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:32.626947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cd8793-e3d2-4059-a2e6-ceab21d21179 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.628179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cd8793-e3d2-4059-a2e6-ceab21d21179 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.628682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cd8793-e3d2-4059-a2e6-ceab21d21179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.629502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 778/1540] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Sep 15 14:09:32.638862 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a9d42c-e900-485c-ba3f-5bd887314538 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.639183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a9d42c-e900-485c-ba3f-5bd887314538 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.639424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a9d42c-e900-485c-ba3f-5bd887314538 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.663193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b4cd8793-e3d2-4059-a2e6-ceab21d21179 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:32.664836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b4cd8793-e3d2-4059-a2e6-ceab21d21179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:32.671433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-68a9d42c-e900-485c-ba3f-5bd887314538 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:32.671433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-68a9d42c-e900-485c-ba3f-5bd887314538 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:32.683310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 763/1541] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:32.690700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 779/1542] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:32.695852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cd8793-e3d2-4059-a2e6-ceab21d21179 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:32.702387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f8da1-79c1-4459-b500-21860b56cde1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.702676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f8da1-79c1-4459-b500-21860b56cde1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.702951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f8da1-79c1-4459-b500-21860b56cde1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.710724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cd8793-e3d2-4059-a2e6-ceab21d21179 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:32.719146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d72977-5de3-494b-915c-78dd51859a1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:32.720671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d72977-5de3-494b-915c-78dd51859a1d admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.721340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d72977-5de3-494b-915c-78dd51859a1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.721850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d72977-5de3-494b-915c-78dd51859a1d admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.722848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d72977-5de3-494b-915c-78dd51859a1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:32.723375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d72977-5de3-494b-915c-78dd51859a1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:32.728704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING py.warnings [None req-f0d72977-5de3-494b-915c-78dd51859a1d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:09:32.728704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: warnings.warn(msg) Sep 15 14:09:32.728704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: Sep 15 14:09:32.730092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d72977-5de3-494b-915c-78dd51859a1d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:32.734928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-824f8da1-79c1-4459-b500-21860b56cde1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:32.736189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-824f8da1-79c1-4459-b500-21860b56cde1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:32.748600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 764/1543] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:32.750278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968ef87-6574-4c49-ac45-b942c4117335 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.750972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968ef87-6574-4c49-ac45-b942c4117335 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.751476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968ef87-6574-4c49-ac45-b942c4117335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.761079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 780/1544] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:32.761968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f8da1-79c1-4459-b500-21860b56cde1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:32.784862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2968ef87-6574-4c49-ac45-b942c4117335 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:32.786513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2968ef87-6574-4c49-ac45-b942c4117335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:32.790522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824f8da1-79c1-4459-b500-21860b56cde1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:32.798589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 765/1545] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:32.801135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2968ef87-6574-4c49-ac45-b942c4117335 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:32.805441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c227e28f-e54d-4183-980c-e7897a71ae89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:32.806077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c227e28f-e54d-4183-980c-e7897a71ae89 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.806355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c227e28f-e54d-4183-980c-e7897a71ae89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.806884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c227e28f-e54d-4183-980c-e7897a71ae89 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.807524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c227e28f-e54d-4183-980c-e7897a71ae89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:32.808656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING py.warnings [None req-c227e28f-e54d-4183-980c-e7897a71ae89 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:09:32.808656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: warnings.warn(msg) Sep 15 14:09:32.808656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: Sep 15 14:09:32.810283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c227e28f-e54d-4183-980c-e7897a71ae89 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:32.817701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2968ef87-6574-4c49-ac45-b942c4117335 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:32.835183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4021a247-e533-45d8-b116-760e64206b98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:32.836193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4021a247-e533-45d8-b116-760e64206b98 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.836712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4021a247-e533-45d8-b116-760e64206b98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.837210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4021a247-e533-45d8-b116-760e64206b98 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.838245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4021a247-e533-45d8-b116-760e64206b98 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:32.838766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4021a247-e533-45d8-b116-760e64206b98 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:32.840627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4021a247-e533-45d8-b116-760e64206b98 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:32.854993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 766/1546] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:32.856682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d19c7-943a-4e49-a00a-5b7121e5389a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.857205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d19c7-943a-4e49-a00a-5b7121e5389a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.857696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d19c7-943a-4e49-a00a-5b7121e5389a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.877082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 781/1547] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/projects => generated 375 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:32.878536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39ad5c-8b77-472d-a991-f41bf24a91c9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.878817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39ad5c-8b77-472d-a991-f41bf24a91c9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.879088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd39ad5c-8b77-472d-a991-f41bf24a91c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.889855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-eb9d19c7-943a-4e49-a00a-5b7121e5389a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:32.893056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-eb9d19c7-943a-4e49-a00a-5b7121e5389a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:32.900514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-fd39ad5c-8b77-472d-a991-f41bf24a91c9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:32.902154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-fd39ad5c-8b77-472d-a991-f41bf24a91c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:32.955289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 767/1548] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:32.960361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87ac2f-414d-4cfe-bd93-d81bd8c7a0c2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.960652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87ac2f-414d-4cfe-bd93-d81bd8c7a0c2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.960932 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87ac2f-414d-4cfe-bd93-d81bd8c7a0c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.968235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 782/1549] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:32.974074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc9ea10-21a8-4859-931f-6972ffd93c5b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.974364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc9ea10-21a8-4859-931f-6972ffd93c5b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.974670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc9ea10-21a8-4859-931f-6972ffd93c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:32.981525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-8b87ac2f-414d-4cfe-bd93-d81bd8c7a0c2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:32.982842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8b87ac2f-414d-4cfe-bd93-d81bd8c7a0c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:32.992554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 768/1550] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:32.993754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda84fe-86f7-45cd-93df-c42ea528f7dd None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:32.994040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda84fe-86f7-45cd-93df-c42ea528f7dd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:32.994320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda84fe-86f7-45cd-93df-c42ea528f7dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.000349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5fc9ea10-21a8-4859-931f-6972ffd93c5b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:33.001983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5fc9ea10-21a8-4859-931f-6972ffd93c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:33.010010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 783/1551] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:33.010771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc9ea10-21a8-4859-931f-6972ffd93c5b None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.013556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-eeda84fe-86f7-45cd-93df-c42ea528f7dd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:33.014739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-eeda84fe-86f7-45cd-93df-c42ea528f7dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:33.020667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc9ea10-21a8-4859-931f-6972ffd93c5b None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.023434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 769/1552] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:33.024264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda84fe-86f7-45cd-93df-c42ea528f7dd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.026733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ee50a4-15f8-4685-b926-872aa3bb117c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.027336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ee50a4-15f8-4685-b926-872aa3bb117c admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.027610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ee50a4-15f8-4685-b926-872aa3bb117c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.028043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ee50a4-15f8-4685-b926-872aa3bb117c admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.028672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ee50a4-15f8-4685-b926-872aa3bb117c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.032316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ee50a4-15f8-4685-b926-872aa3bb117c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.035023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda84fe-86f7-45cd-93df-c42ea528f7dd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.039597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.040224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.040509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.040783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.041413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.042409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING py.warnings [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:09:33.042409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: warnings.warn(msg) Sep 15 14:09:33.042409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: Sep 15 14:09:33.043628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.061832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 784/1553] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/projects => generated 403 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.065098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81dae36-aa11-456a-912f-8c613d341bc7 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.065616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81dae36-aa11-456a-912f-8c613d341bc7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.065942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81dae36-aa11-456a-912f-8c613d341bc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.066490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 770/1554] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/users => generated 288 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.067320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.077630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4677902b-5d3f-46ff-b011-2fcfa5813d51 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.087427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c81dae36-aa11-456a-912f-8c613d341bc7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:33.088581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c81dae36-aa11-456a-912f-8c613d341bc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:33.135830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afe2ea-6140-416b-9ac7-2cdf2a79e262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.136441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afe2ea-6140-416b-9ac7-2cdf2a79e262 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.136719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afe2ea-6140-416b-9ac7-2cdf2a79e262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.136994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afe2ea-6140-416b-9ac7-2cdf2a79e262 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.137640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afe2ea-6140-416b-9ac7-2cdf2a79e262 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:33.137958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afe2ea-6140-416b-9ac7-2cdf2a79e262 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.139558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afe2ea-6140-416b-9ac7-2cdf2a79e262 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.148540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 785/1555] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:33.149790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c485301-4a9f-4a97-bff1-cd89d9f64ecb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.150078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c485301-4a9f-4a97-bff1-cd89d9f64ecb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.150346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c485301-4a9f-4a97-bff1-cd89d9f64ecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.158716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 771/1556] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:33.160260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79832060-6074-44aa-820d-0a058ae1e973 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.160543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79832060-6074-44aa-820d-0a058ae1e973 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.160818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79832060-6074-44aa-820d-0a058ae1e973 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.177704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2c485301-4a9f-4a97-bff1-cd89d9f64ecb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:33.178902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2c485301-4a9f-4a97-bff1-cd89d9f64ecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:33.180897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-79832060-6074-44aa-820d-0a058ae1e973 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:33.182411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-79832060-6074-44aa-820d-0a058ae1e973 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:33.187273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 786/1557] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:33.188420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db43cd5-0271-46cf-a9ff-f19ec335bb0b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.188706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db43cd5-0271-46cf-a9ff-f19ec335bb0b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.188981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db43cd5-0271-46cf-a9ff-f19ec335bb0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.194470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 772/1558] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:33.194470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7473d3a5-c202-4fc7-be6b-882d38b7118d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.194470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7473d3a5-c202-4fc7-be6b-882d38b7118d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.194470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7473d3a5-c202-4fc7-be6b-882d38b7118d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.212217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-7473d3a5-c202-4fc7-be6b-882d38b7118d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:33.212425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9db43cd5-0271-46cf-a9ff-f19ec335bb0b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:33.213390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9db43cd5-0271-46cf-a9ff-f19ec335bb0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:33.213488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7473d3a5-c202-4fc7-be6b-882d38b7118d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:33.225380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 787/1560] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:33.225546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 773/1560] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:33.225648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db43cd5-0271-46cf-a9ff-f19ec335bb0b None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.225765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7473d3a5-c202-4fc7-be6b-882d38b7118d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.234946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7473d3a5-c202-4fc7-be6b-882d38b7118d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.235485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db43cd5-0271-46cf-a9ff-f19ec335bb0b None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.239782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4a80aa-1cfc-408f-876d-db0de4e21c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.240080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81312c18-6a6d-4009-8689-68e8cf4fcd5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.240368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a80aa-1cfc-408f-876d-db0de4e21c9b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.240646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a80aa-1cfc-408f-876d-db0de4e21c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.240772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81312c18-6a6d-4009-8689-68e8cf4fcd5f admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.241009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4a80aa-1cfc-408f-876d-db0de4e21c9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.241112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81312c18-6a6d-4009-8689-68e8cf4fcd5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.241271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81312c18-6a6d-4009-8689-68e8cf4fcd5f admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.241898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81312c18-6a6d-4009-8689-68e8cf4fcd5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.242155 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4a80aa-1cfc-408f-876d-db0de4e21c9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.243858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81312c18-6a6d-4009-8689-68e8cf4fcd5f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.245176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4a80aa-1cfc-408f-876d-db0de4e21c9b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.252289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:33.253309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:33.257547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 774/1561] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:33.259650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd50d5-de9d-48e9-880b-4a480a20e432 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.260021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd50d5-de9d-48e9-880b-4a480a20e432 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.260724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd50d5-de9d-48e9-880b-4a480a20e432 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.270086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 788/1562] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/users => generated 302 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.272153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81312c18-6a6d-4009-8689-68e8cf4fcd5f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.281912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81312c18-6a6d-4009-8689-68e8cf4fcd5f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.283610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-45fd50d5-de9d-48e9-880b-4a480a20e432 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:33.284931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-45fd50d5-de9d-48e9-880b-4a480a20e432 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:33.286773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d44e60e-a0ee-4adf-9018-24783741d410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.287364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d44e60e-a0ee-4adf-9018-24783741d410 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.287640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d44e60e-a0ee-4adf-9018-24783741d410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.287938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d44e60e-a0ee-4adf-9018-24783741d410 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.288648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d44e60e-a0ee-4adf-9018-24783741d410 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.290496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d44e60e-a0ee-4adf-9018-24783741d410 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.293078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 775/1563] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:33.293857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd50d5-de9d-48e9-880b-4a480a20e432 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.304144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd50d5-de9d-48e9-880b-4a480a20e432 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.310151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.311304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.311304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.311841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.312508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:33.312825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.315235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.327334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 789/1564] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/projects => generated 387 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.328541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d44e60e-a0ee-4adf-9018-24783741d410 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.331758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 776/1565] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:33.332794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.343373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e7472-7ce8-4832-a389-5ff9fc2be440 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.394026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d44e60e-a0ee-4adf-9018-24783741d410 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.398607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.399211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.399493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.399797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.400448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:33.400748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.401104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.401716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.402062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.402262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.402375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.402908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:33.403210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.404860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.414245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 790/1566] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:33.415325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.424345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 777/1567] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:33.425197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.428689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d9f11-2996-41cc-8b1a-9b3892e88b4e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.434804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.434941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.435174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.435587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.437577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:33.437894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.439644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4db570-a108-482c-8f23-b44b7b712339 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.440877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.448370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1505480-70a5-43e9-a4ee-bbfc9db41445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.449005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1505480-70a5-43e9-a4ee-bbfc9db41445 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.449687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1505480-70a5-43e9-a4ee-bbfc9db41445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.449687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1505480-70a5-43e9-a4ee-bbfc9db41445 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.450330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1505480-70a5-43e9-a4ee-bbfc9db41445 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.452006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1505480-70a5-43e9-a4ee-bbfc9db41445 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.454836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 791/1568] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:33.455624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.466030 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 778/1569] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:33.466030 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1505480-70a5-43e9-a4ee-bbfc9db41445 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.466345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93581c1-ea34-46d5-92eb-7f5dc8bbcd8d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.471483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a4742d-0fc9-438f-83c1-a8867207d4c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.472316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a4742d-0fc9-438f-83c1-a8867207d4c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.472476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a4742d-0fc9-438f-83c1-a8867207d4c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.472756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a4742d-0fc9-438f-83c1-a8867207d4c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.473473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a4742d-0fc9-438f-83c1-a8867207d4c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.474640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1505480-70a5-43e9-a4ee-bbfc9db41445 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.475111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a4742d-0fc9-438f-83c1-a8867207d4c9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.479381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.480013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.480294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.480704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.481375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:33.481678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.483340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.486401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 792/1570] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:33.487121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a4742d-0fc9-438f-83c1-a8867207d4c9 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.497523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a4742d-0fc9-438f-83c1-a8867207d4c9 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.499123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 779/1571] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:33.501358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.503241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb05bd-0fcd-414f-bd88-bc48f54fe534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.503878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb05bd-0fcd-414f-bd88-bc48f54fe534 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.504165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb05bd-0fcd-414f-bd88-bc48f54fe534 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.504441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb05bd-0fcd-414f-bd88-bc48f54fe534 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.505077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddb05bd-0fcd-414f-bd88-bc48f54fe534 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.507280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddb05bd-0fcd-414f-bd88-bc48f54fe534 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.511613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bacdca-236d-418b-8f21-85cdbc9d6b5e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.516938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe4910c-53b4-4f54-93f5-67189db5359e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.517911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4910c-53b4-4f54-93f5-67189db5359e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.518508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4910c-53b4-4f54-93f5-67189db5359e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.518508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4910c-53b4-4f54-93f5-67189db5359e admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.519097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe4910c-53b4-4f54-93f5-67189db5359e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.521353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe4910c-53b4-4f54-93f5-67189db5359e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.529008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 793/1572] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/users => generated 294 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.530147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb05bd-0fcd-414f-bd88-bc48f54fe534 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.542371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb05bd-0fcd-414f-bd88-bc48f54fe534 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.546963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6462f357-dd8d-47d5-a540-1ef848d79f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.548106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6462f357-dd8d-47d5-a540-1ef848d79f9d admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.548385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6462f357-dd8d-47d5-a540-1ef848d79f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.548659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6462f357-dd8d-47d5-a540-1ef848d79f9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.552243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 780/1573] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.552399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6462f357-dd8d-47d5-a540-1ef848d79f9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.552508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe4910c-53b4-4f54-93f5-67189db5359e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.552603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6462f357-dd8d-47d5-a540-1ef848d79f9d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.580917 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 794/1574] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/projects => generated 387 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.584703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6462f357-dd8d-47d5-a540-1ef848d79f9d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.617527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe4910c-53b4-4f54-93f5-67189db5359e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.648366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6462f357-dd8d-47d5-a540-1ef848d79f9d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.652640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:33.653202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0f284b-266e-41e2-affe-6b87bb3b0868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.653675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0f284b-266e-41e2-affe-6b87bb3b0868 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.653954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0f284b-266e-41e2-affe-6b87bb3b0868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.654232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0f284b-266e-41e2-affe-6b87bb3b0868 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.654867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0f284b-266e-41e2-affe-6b87bb3b0868 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.656896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0f284b-266e-41e2-affe-6b87bb3b0868 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.661700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:33.679050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72e1883-8458-42ad-b0e3-f53216492fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.679722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72e1883-8458-42ad-b0e3-f53216492fd4 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.680231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72e1883-8458-42ad-b0e3-f53216492fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.680365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72e1883-8458-42ad-b0e3-f53216492fd4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.680900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72e1883-8458-42ad-b0e3-f53216492fd4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.683165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72e1883-8458-42ad-b0e3-f53216492fd4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.686066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 795/1575] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/projects => generated 393 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.686877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0f284b-266e-41e2-affe-6b87bb3b0868 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.713437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 781/1576] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/projects => generated 371 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.714555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72e1883-8458-42ad-b0e3-f53216492fd4 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.748782 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0f284b-266e-41e2-affe-6b87bb3b0868 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.781077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72e1883-8458-42ad-b0e3-f53216492fd4 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.786037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2622bbab-8e0f-4f7d-bb76-b18856a8552d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.786719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2622bbab-8e0f-4f7d-bb76-b18856a8552d admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.787062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2622bbab-8e0f-4f7d-bb76-b18856a8552d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.787401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2622bbab-8e0f-4f7d-bb76-b18856a8552d admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.788141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2622bbab-8e0f-4f7d-bb76-b18856a8552d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.789815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2622bbab-8e0f-4f7d-bb76-b18856a8552d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.803202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 782/1577] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:33.803202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2622bbab-8e0f-4f7d-bb76-b18856a8552d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.804431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.805102 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.805385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.805770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] PATH_INFO: `/v3/projects/a8297077982a4338a2d687c89e1119a2/users/2049d66708ce47a794c14ad4c2040937/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.811600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2622bbab-8e0f-4f7d-bb76-b18856a8552d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.816700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a067c23c-d0ba-4cc8-a014-48e154317c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.817299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067c23c-d0ba-4cc8-a014-48e154317c6e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.817578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067c23c-d0ba-4cc8-a014-48e154317c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.817857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067c23c-d0ba-4cc8-a014-48e154317c6e admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.818491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a067c23c-d0ba-4cc8-a014-48e154317c6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.820428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a067c23c-d0ba-4cc8-a014-48e154317c6e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.822513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8297077982a4338a2d687c89e1119a2, user_id=2049d66708ce47a794c14ad4c2040937, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.823539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING py.warnings [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:09:33.823539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: warnings.warn(msg) Sep 15 14:09:33.823539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: Sep 15 14:09:33.825521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.839013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 796/1578] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:33 2022] PUT /identity/v3/projects/a8297077982a4338a2d687c89e1119a2/users/2049d66708ce47a794c14ad4c2040937/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:33.840694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.855727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 783/1579] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/projects => generated 377 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.855727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a067c23c-d0ba-4cc8-a014-48e154317c6e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.902884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5d08b9-f340-4c7f-8e8d-c0347339487c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.917000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a067c23c-d0ba-4cc8-a014-48e154317c6e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.920316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:33.921876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e363096f-9245-4378-a388-b7e688eaa0ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.922619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e363096f-9245-4378-a388-b7e688eaa0ee admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.923017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e363096f-9245-4378-a388-b7e688eaa0ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.923351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e363096f-9245-4378-a388-b7e688eaa0ee admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.924035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e363096f-9245-4378-a388-b7e688eaa0ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.926006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e363096f-9245-4378-a388-b7e688eaa0ee admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.951787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 784/1580] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/users => generated 287 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.951787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e363096f-9245-4378-a388-b7e688eaa0ee admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.960347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e363096f-9245-4378-a388-b7e688eaa0ee admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.965083 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5112b48a-2b71-49b9-ae67-8e902f2510e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.965673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5112b48a-2b71-49b9-ae67-8e902f2510e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.965981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5112b48a-2b71-49b9-ae67-8e902f2510e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.966255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5112b48a-2b71-49b9-ae67-8e902f2510e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.966873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5112b48a-2b71-49b9-ae67-8e902f2510e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.967929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1199c259-9f9b-4c92-867b-d823a3c9322b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.968569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199c259-9f9b-4c92-867b-d823a3c9322b admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.968820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5112b48a-2b71-49b9-ae67-8e902f2510e2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.968930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199c259-9f9b-4c92-867b-d823a3c9322b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.969623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199c259-9f9b-4c92-867b-d823a3c9322b admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:33.970278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1199c259-9f9b-4c92-867b-d823a3c9322b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:33.971887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1199c259-9f9b-4c92-867b-d823a3c9322b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:33.982144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 797/1581] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:33 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:33.983193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1199c259-9f9b-4c92-867b-d823a3c9322b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.993059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 785/1582] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/users => generated 294 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:33.993849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5112b48a-2b71-49b9-ae67-8e902f2510e2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:33.993966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1199c259-9f9b-4c92-867b-d823a3c9322b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:33.998518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dc31f6-5cee-45a5-91a6-afd3a52c85b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:33.999113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dc31f6-5cee-45a5-91a6-afd3a52c85b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:33.999392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dc31f6-5cee-45a5-91a6-afd3a52c85b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:33.999665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dc31f6-5cee-45a5-91a6-afd3a52c85b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.000326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dc31f6-5cee-45a5-91a6-afd3a52c85b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.003088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dc31f6-5cee-45a5-91a6-afd3a52c85b4 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.004132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5112b48a-2b71-49b9-ae67-8e902f2510e2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.008757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be32acaf-a889-4a1f-b574-9ccc0a7e76fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.009341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be32acaf-a889-4a1f-b574-9ccc0a7e76fb admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.009620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be32acaf-a889-4a1f-b574-9ccc0a7e76fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.009896 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be32acaf-a889-4a1f-b574-9ccc0a7e76fb admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.010515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be32acaf-a889-4a1f-b574-9ccc0a7e76fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.012454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be32acaf-a889-4a1f-b574-9ccc0a7e76fb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.020522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:34.023756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 798/1583] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/users => generated 297 bytes in 41 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:34.024485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dc31f6-5cee-45a5-91a6-afd3a52c85b4 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.034894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dc31f6-5cee-45a5-91a6-afd3a52c85b4 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.040906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 786/1584] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:33 2022] POST /identity/v3/users => generated 286 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:34.040906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97ecfb0-91fe-4de5-af49-0eb86374b3e6 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.040906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97ecfb0-91fe-4de5-af49-0eb86374b3e6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.040906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97ecfb0-91fe-4de5-af49-0eb86374b3e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.042930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0030218f-090d-4568-ab93-c28b9f6dfb26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.043586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0030218f-090d-4568-ab93-c28b9f6dfb26 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.043890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0030218f-090d-4568-ab93-c28b9f6dfb26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.044168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0030218f-090d-4568-ab93-c28b9f6dfb26 admin admin] PATH_INFO: `/v3/projects/a84b43c2ba1f4497858e2be943acf2d4/users/09e7e3f3c2a84a1aa1e16324312a57f6/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.058733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b97ecfb0-91fe-4de5-af49-0eb86374b3e6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:34.059970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0030218f-090d-4568-ab93-c28b9f6dfb26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a84b43c2ba1f4497858e2be943acf2d4, user_id=09e7e3f3c2a84a1aa1e16324312a57f6, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.060085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b97ecfb0-91fe-4de5-af49-0eb86374b3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:34.062640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0030218f-090d-4568-ab93-c28b9f6dfb26 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.068353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 787/1585] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:34.069178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97ecfb0-91fe-4de5-af49-0eb86374b3e6 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.077513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 799/1586] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:34 2022] PUT /identity/v3/projects/a84b43c2ba1f4497858e2be943acf2d4/users/09e7e3f3c2a84a1aa1e16324312a57f6/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:34.079084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97ecfb0-91fe-4de5-af49-0eb86374b3e6 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.085146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0030218f-090d-4568-ab93-c28b9f6dfb26 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.135815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f01d05-3502-45e5-87c4-0a6cddffbf8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.136425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f01d05-3502-45e5-87c4-0a6cddffbf8d admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.136707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f01d05-3502-45e5-87c4-0a6cddffbf8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.137020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f01d05-3502-45e5-87c4-0a6cddffbf8d admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.137618 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f01d05-3502-45e5-87c4-0a6cddffbf8d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.139541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f01d05-3502-45e5-87c4-0a6cddffbf8d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.147170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0030218f-090d-4568-ab93-c28b9f6dfb26 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.153159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d840bc-f3ad-45ec-9c2d-410d9eaac91c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.153686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d840bc-f3ad-45ec-9c2d-410d9eaac91c admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.154028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d840bc-f3ad-45ec-9c2d-410d9eaac91c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.154329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d840bc-f3ad-45ec-9c2d-410d9eaac91c admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.155070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d840bc-f3ad-45ec-9c2d-410d9eaac91c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.156869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d840bc-f3ad-45ec-9c2d-410d9eaac91c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.162434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 788/1587] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:34 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:34.163085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f01d05-3502-45e5-87c4-0a6cddffbf8d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.174014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 800/1588] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.174014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d840bc-f3ad-45ec-9c2d-410d9eaac91c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.174308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f01d05-3502-45e5-87c4-0a6cddffbf8d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.183123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ac6c3-bfbc-4af5-91a7-4756a121458e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.183777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ac6c3-bfbc-4af5-91a7-4756a121458e admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.183920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d840bc-f3ad-45ec-9c2d-410d9eaac91c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.184116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ac6c3-bfbc-4af5-91a7-4756a121458e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.184441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ac6c3-bfbc-4af5-91a7-4756a121458e admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.185129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ac6c3-bfbc-4af5-91a7-4756a121458e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.186755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ac6c3-bfbc-4af5-91a7-4756a121458e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.189397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dcef98-ecb3-4d79-98dc-6d08b3c14d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.189985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dcef98-ecb3-4d79-98dc-6d08b3c14d58 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.190266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dcef98-ecb3-4d79-98dc-6d08b3c14d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.190543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dcef98-ecb3-4d79-98dc-6d08b3c14d58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.191187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dcef98-ecb3-4d79-98dc-6d08b3c14d58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.192812 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dcef98-ecb3-4d79-98dc-6d08b3c14d58 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.198168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 789/1589] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.198967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ac6c3-bfbc-4af5-91a7-4756a121458e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.206335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 801/1590] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.207035 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dcef98-ecb3-4d79-98dc-6d08b3c14d58 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.209196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ac6c3-bfbc-4af5-91a7-4756a121458e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.213736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3731fa-c5fb-4d39-90a7-2ece1ba1ec18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.214328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3731fa-c5fb-4d39-90a7-2ece1ba1ec18 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.214611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3731fa-c5fb-4d39-90a7-2ece1ba1ec18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.214887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3731fa-c5fb-4d39-90a7-2ece1ba1ec18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.215521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3731fa-c5fb-4d39-90a7-2ece1ba1ec18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.216878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dcef98-ecb3-4d79-98dc-6d08b3c14d58 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.217118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3731fa-c5fb-4d39-90a7-2ece1ba1ec18 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.221584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c109548-bd2d-41ce-85b7-8940413f7a54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.222165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c109548-bd2d-41ce-85b7-8940413f7a54 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.222453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c109548-bd2d-41ce-85b7-8940413f7a54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.222735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c109548-bd2d-41ce-85b7-8940413f7a54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.223367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c109548-bd2d-41ce-85b7-8940413f7a54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.224990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c109548-bd2d-41ce-85b7-8940413f7a54 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.230656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 790/1591] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.230656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3731fa-c5fb-4d39-90a7-2ece1ba1ec18 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.238985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 802/1592] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.238985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7286b63-b2ce-4413-9504-93461bea528a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.238985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7286b63-b2ce-4413-9504-93461bea528a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.238985 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7286b63-b2ce-4413-9504-93461bea528a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.265541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d7286b63-b2ce-4413-9504-93461bea528a None None] MFA Rules not processed for user `09e7e3f3c2a84a1aa1e16324312a57f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:34.266779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d7286b63-b2ce-4413-9504-93461bea528a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:34.286106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3731fa-c5fb-4d39-90a7-2ece1ba1ec18 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.290035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590730b-e80d-4de0-abc4-a1470653903f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.290639 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590730b-e80d-4de0-abc4-a1470653903f service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.290930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590730b-e80d-4de0-abc4-a1470653903f service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.291198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590730b-e80d-4de0-abc4-a1470653903f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.301056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4590730b-e80d-4de0-abc4-a1470653903f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.303295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4590730b-e80d-4de0-abc4-a1470653903f service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.309209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:34.317204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 791/1593] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:34.318046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590730b-e80d-4de0-abc4-a1470653903f service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.327189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590730b-e80d-4de0-abc4-a1470653903f service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.332479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844a123-248b-4abc-a4da-f34b3236032f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.333084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844a123-248b-4abc-a4da-f34b3236032f admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.333367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844a123-248b-4abc-a4da-f34b3236032f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.333642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844a123-248b-4abc-a4da-f34b3236032f admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.334284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1844a123-248b-4abc-a4da-f34b3236032f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.335921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1844a123-248b-4abc-a4da-f34b3236032f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.342415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 803/1594] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:34 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:34.345028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7286b63-b2ce-4413-9504-93461bea528a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.352575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 792/1595] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.352575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844a123-248b-4abc-a4da-f34b3236032f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.357544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7286b63-b2ce-4413-9504-93461bea528a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.362749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844a123-248b-4abc-a4da-f34b3236032f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.366194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d130e-583b-48c3-81a5-e9527c0d664d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.367049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d130e-583b-48c3-81a5-e9527c0d664d admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.367255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d130e-583b-48c3-81a5-e9527c0d664d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.367715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d130e-583b-48c3-81a5-e9527c0d664d admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.368815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb639957-8b88-4707-bde4-49d02a6f0a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.368965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17d130e-583b-48c3-81a5-e9527c0d664d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.369662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb639957-8b88-4707-bde4-49d02a6f0a00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.369899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb639957-8b88-4707-bde4-49d02a6f0a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.370096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb639957-8b88-4707-bde4-49d02a6f0a00 admin admin] PATH_INFO: `/v3/projects/e76574d5f2094997a9c565d4ddf2a483/users/00374d3d3bf649749781792273e35e3b/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.372031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17d130e-583b-48c3-81a5-e9527c0d664d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.385367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 804/1596] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.386121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d130e-583b-48c3-81a5-e9527c0d664d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.390127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb639957-8b88-4707-bde4-49d02a6f0a00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e76574d5f2094997a9c565d4ddf2a483, user_id=00374d3d3bf649749781792273e35e3b, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.393152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb639957-8b88-4707-bde4-49d02a6f0a00 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.397062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d130e-583b-48c3-81a5-e9527c0d664d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.402768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b593d03-5892-4257-b856-6914a6a6e65f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.403909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b593d03-5892-4257-b856-6914a6a6e65f admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.404386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b593d03-5892-4257-b856-6914a6a6e65f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.404682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b593d03-5892-4257-b856-6914a6a6e65f admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.405321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b593d03-5892-4257-b856-6914a6a6e65f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.407980 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b593d03-5892-4257-b856-6914a6a6e65f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.411360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 793/1597] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:34 2022] PUT /identity/v3/projects/e76574d5f2094997a9c565d4ddf2a483/users/00374d3d3bf649749781792273e35e3b/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:34.412199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb639957-8b88-4707-bde4-49d02a6f0a00 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.424710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 805/1598] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.425776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b593d03-5892-4257-b856-6914a6a6e65f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.436919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b593d03-5892-4257-b856-6914a6a6e65f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.441806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:34.492957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb639957-8b88-4707-bde4-49d02a6f0a00 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.498459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5316c496-88e7-490f-9341-ea227242175b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.499204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5316c496-88e7-490f-9341-ea227242175b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.499508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5316c496-88e7-490f-9341-ea227242175b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.499812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5316c496-88e7-490f-9341-ea227242175b admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.500471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5316c496-88e7-490f-9341-ea227242175b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.502049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5316c496-88e7-490f-9341-ea227242175b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.504700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccba7e7-ba1b-49a0-8792-9a811285beba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.505945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccba7e7-ba1b-49a0-8792-9a811285beba admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.506306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccba7e7-ba1b-49a0-8792-9a811285beba admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.506655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccba7e7-ba1b-49a0-8792-9a811285beba admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.507408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccba7e7-ba1b-49a0-8792-9a811285beba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.510061 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccba7e7-ba1b-49a0-8792-9a811285beba admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.513273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 794/1599] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.517402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5316c496-88e7-490f-9341-ea227242175b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.529315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 806/1600] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.529315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccba7e7-ba1b-49a0-8792-9a811285beba admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.529890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5316c496-88e7-490f-9341-ea227242175b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.537581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aba7ba9-31e7-4aef-984d-807913377e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.538474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aba7ba9-31e7-4aef-984d-807913377e4c admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.539317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aba7ba9-31e7-4aef-984d-807913377e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.539317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aba7ba9-31e7-4aef-984d-807913377e4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.540169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aba7ba9-31e7-4aef-984d-807913377e4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.543371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aba7ba9-31e7-4aef-984d-807913377e4c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.546461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccba7e7-ba1b-49a0-8792-9a811285beba admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.552406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6d7bb7-e48a-477f-b1d1-d26ca78c8334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.553333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d7bb7-e48a-477f-b1d1-d26ca78c8334 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.553620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d7bb7-e48a-477f-b1d1-d26ca78c8334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.553901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d7bb7-e48a-477f-b1d1-d26ca78c8334 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.555488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6d7bb7-e48a-477f-b1d1-d26ca78c8334 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.556351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6d7bb7-e48a-477f-b1d1-d26ca78c8334 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.570068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 807/1601] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:34.570745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6d7bb7-e48a-477f-b1d1-d26ca78c8334 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.580814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 795/1602] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:34 2022] POST /identity/v3/projects => generated 403 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:34.581442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6d7bb7-e48a-477f-b1d1-d26ca78c8334 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.585455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7025-1629-463a-bb6c-45ee26936018 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.585836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7025-1629-463a-bb6c-45ee26936018 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.586183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7025-1629-463a-bb6c-45ee26936018 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.606968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-0a9b7025-1629-463a-bb6c-45ee26936018 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:34.608404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0a9b7025-1629-463a-bb6c-45ee26936018 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:34.652778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aedce4f-0fc9-4dd7-81ab-fb5c9a9a1924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.653450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aedce4f-0fc9-4dd7-81ab-fb5c9a9a1924 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.653733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aedce4f-0fc9-4dd7-81ab-fb5c9a9a1924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.654063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aedce4f-0fc9-4dd7-81ab-fb5c9a9a1924 admin admin] PATH_INFO: `/v3/projects/96b37cc8c62e4096b072abff9b051def/users/d9baa62cc65147719248fe7e898dbb8d/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.670244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aedce4f-0fc9-4dd7-81ab-fb5c9a9a1924 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96b37cc8c62e4096b072abff9b051def, user_id=d9baa62cc65147719248fe7e898dbb8d, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.673490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aedce4f-0fc9-4dd7-81ab-fb5c9a9a1924 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.680508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 796/1603] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:34.680508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b7025-1629-463a-bb6c-45ee26936018 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.689842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b7025-1629-463a-bb6c-45ee26936018 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.699780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 808/1604] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:34 2022] PUT /identity/v3/projects/96b37cc8c62e4096b072abff9b051def/users/d9baa62cc65147719248fe7e898dbb8d/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:34.701378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aedce4f-0fc9-4dd7-81ab-fb5c9a9a1924 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.745544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:34.753523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a590f79-dfd1-40c6-b023-d47abb10169a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.754437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a590f79-dfd1-40c6-b023-d47abb10169a admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.754815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a590f79-dfd1-40c6-b023-d47abb10169a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.755154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a590f79-dfd1-40c6-b023-d47abb10169a admin admin] PATH_INFO: `/v3/projects/3f927dff41404c4ba81b5d8882c99dd4/users/90d193d31f0c43999fdb052d28e7d833/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.774397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a590f79-dfd1-40c6-b023-d47abb10169a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f927dff41404c4ba81b5d8882c99dd4, user_id=90d193d31f0c43999fdb052d28e7d833, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.777299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a590f79-dfd1-40c6-b023-d47abb10169a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.783539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aedce4f-0fc9-4dd7-81ab-fb5c9a9a1924 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.793680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62783ab3-1046-46bd-b8b0-29e4a658dbfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.794464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 797/1605] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:34 2022] PUT /identity/v3/projects/3f927dff41404c4ba81b5d8882c99dd4/users/90d193d31f0c43999fdb052d28e7d833/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:34.795004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62783ab3-1046-46bd-b8b0-29e4a658dbfc admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.795283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a590f79-dfd1-40c6-b023-d47abb10169a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.795817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62783ab3-1046-46bd-b8b0-29e4a658dbfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.796383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62783ab3-1046-46bd-b8b0-29e4a658dbfc admin admin] PATH_INFO: `/v3/projects/fab43b244e854c0296530db48bc1b975/users/530e6587b39b4f4c98d7414b81453628/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.815516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62783ab3-1046-46bd-b8b0-29e4a658dbfc admin admin] RBAC: Authorizing `identity:create_grant(project_id=fab43b244e854c0296530db48bc1b975, user_id=530e6587b39b4f4c98d7414b81453628, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.818977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62783ab3-1046-46bd-b8b0-29e4a658dbfc admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.826971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:34.843439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 809/1606] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:34 2022] PUT /identity/v3/projects/fab43b244e854c0296530db48bc1b975/users/530e6587b39b4f4c98d7414b81453628/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:34.845311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62783ab3-1046-46bd-b8b0-29e4a658dbfc admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.885153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a590f79-dfd1-40c6-b023-d47abb10169a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.919033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62783ab3-1046-46bd-b8b0-29e4a658dbfc admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:34.924103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561c1bb1-e508-4ced-8ce7-c95eef105b12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.924789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561c1bb1-e508-4ced-8ce7-c95eef105b12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.925067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561c1bb1-e508-4ced-8ce7-c95eef105b12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.925546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561c1bb1-e508-4ced-8ce7-c95eef105b12 admin admin] PATH_INFO: `/v3/projects/d903de84a9c7446e9a2f368e5eab8719/users/f521b038310940d1b5ee7fc5afc1cbac/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.943822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561c1bb1-e508-4ced-8ce7-c95eef105b12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d903de84a9c7446e9a2f368e5eab8719, user_id=f521b038310940d1b5ee7fc5afc1cbac, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.946978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561c1bb1-e508-4ced-8ce7-c95eef105b12 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.957599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:34.958385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:34.958657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:34.958964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] PATH_INFO: `/v3/projects/08233653cc71488393a34f602b7fd885/users/22bd431202dc4bb7a3e2741a7c1a51b8/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:34.965066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 810/1607] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:34 2022] PUT /identity/v3/projects/d903de84a9c7446e9a2f368e5eab8719/users/f521b038310940d1b5ee7fc5afc1cbac/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:34.965066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561c1bb1-e508-4ced-8ce7-c95eef105b12 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:34.979553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=08233653cc71488393a34f602b7fd885, user_id=22bd431202dc4bb7a3e2741a7c1a51b8, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:34.983535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:34.998868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 798/1608] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:34 2022] PUT /identity/v3/projects/08233653cc71488393a34f602b7fd885/users/22bd431202dc4bb7a3e2741a7c1a51b8/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:35.000007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.002207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:35.054650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561c1bb1-e508-4ced-8ce7-c95eef105b12 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.071167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5d83d-6d1a-4f62-b429-4a35b35f0bfb admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.074786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d39656-5b5a-403a-9fbe-c7bf851d56f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.075386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d39656-5b5a-403a-9fbe-c7bf851d56f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.075670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d39656-5b5a-403a-9fbe-c7bf851d56f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.075996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d39656-5b5a-403a-9fbe-c7bf851d56f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.107927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:35.135972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef53217-9ef6-48fc-97d8-828615db1cbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.136587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef53217-9ef6-48fc-97d8-828615db1cbf admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.136873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef53217-9ef6-48fc-97d8-828615db1cbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.137180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef53217-9ef6-48fc-97d8-828615db1cbf admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.137978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef53217-9ef6-48fc-97d8-828615db1cbf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.139934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef53217-9ef6-48fc-97d8-828615db1cbf admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.160251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:35.164599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d39656-5b5a-403a-9fbe-c7bf851d56f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.166870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d39656-5b5a-403a-9fbe-c7bf851d56f5 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.170726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 811/1609] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:34 2022] POST /identity/v3/users => generated 308 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:35.170726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436a5770-8500-4b5c-a667-524217de5b63 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.170726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436a5770-8500-4b5c-a667-524217de5b63 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.170726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436a5770-8500-4b5c-a667-524217de5b63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.190009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 799/1610] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:35.191367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4096b45-e059-4802-b402-161138ab60b7 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.191794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4096b45-e059-4802-b402-161138ab60b7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.192084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4096b45-e059-4802-b402-161138ab60b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.209597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-436a5770-8500-4b5c-a667-524217de5b63 None None] MFA Rules not processed for user `d9baa62cc65147719248fe7e898dbb8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:35.210813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-436a5770-8500-4b5c-a667-524217de5b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:35.234520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d4096b45-e059-4802-b402-161138ab60b7 None None] MFA Rules not processed for user `90d193d31f0c43999fdb052d28e7d833`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:35.236045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d4096b45-e059-4802-b402-161138ab60b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:35.313656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 812/1611] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:35.313656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e410684-92fa-424d-8aa8-413eead4fc64 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.313656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e410684-92fa-424d-8aa8-413eead4fc64 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.313656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e410684-92fa-424d-8aa8-413eead4fc64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.325186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 800/1612] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:35.332783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c85a2-b889-4af2-9ba6-a0b8ce137f6b None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.333112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c85a2-b889-4af2-9ba6-a0b8ce137f6b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.333630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c85a2-b889-4af2-9ba6-a0b8ce137f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.360763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-bc7c85a2-b889-4af2-9ba6-a0b8ce137f6b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:35.362466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bc7c85a2-b889-4af2-9ba6-a0b8ce137f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:35.367804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-0e410684-92fa-424d-8aa8-413eead4fc64 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:35.372268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 801/1613] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:35.373777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af2231b-a660-41a7-acf0-9d00f4409195 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.374062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af2231b-a660-41a7-acf0-9d00f4409195 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.374332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af2231b-a660-41a7-acf0-9d00f4409195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.382358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0e410684-92fa-424d-8aa8-413eead4fc64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:35.396760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 813/1614] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:35.397531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e410684-92fa-424d-8aa8-413eead4fc64 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.413016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-5af2231b-a660-41a7-acf0-9d00f4409195 None None] MFA Rules not processed for user `22bd431202dc4bb7a3e2741a7c1a51b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:35.414496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5af2231b-a660-41a7-acf0-9d00f4409195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:35.416833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e410684-92fa-424d-8aa8-413eead4fc64 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.424640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4302b7b-ec20-4e2f-8cb6-6952ef9e0e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.425437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4302b7b-ec20-4e2f-8cb6-6952ef9e0e8f admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.426027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4302b7b-ec20-4e2f-8cb6-6952ef9e0e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.426366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4302b7b-ec20-4e2f-8cb6-6952ef9e0e8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.427116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4302b7b-ec20-4e2f-8cb6-6952ef9e0e8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.428888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4302b7b-ec20-4e2f-8cb6-6952ef9e0e8f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.445349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 814/1615] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:35 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:35.446890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4302b7b-ec20-4e2f-8cb6-6952ef9e0e8f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.462992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4302b7b-ec20-4e2f-8cb6-6952ef9e0e8f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.468482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de05d32-a4a0-48fa-b31a-3f475124d9d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.469450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de05d32-a4a0-48fa-b31a-3f475124d9d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.469836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de05d32-a4a0-48fa-b31a-3f475124d9d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.470340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de05d32-a4a0-48fa-b31a-3f475124d9d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.471085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de05d32-a4a0-48fa-b31a-3f475124d9d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.474074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de05d32-a4a0-48fa-b31a-3f475124d9d5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.502630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 802/1616] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:35.508995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deba9d3-2911-4fa6-9e4c-f61f023624ce None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.509449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deba9d3-2911-4fa6-9e4c-f61f023624ce None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.509799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deba9d3-2911-4fa6-9e4c-f61f023624ce None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.511640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 803/1617] 10.222.0.27 () {54 vars in 962 bytes} [Thu Sep 15 14:09:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:09:35.512565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5deba9d3-2911-4fa6-9e4c-f61f023624ce None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.517846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 815/1618] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/projects => generated 387 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:35.518906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de05d32-a4a0-48fa-b31a-3f475124d9d5 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.596458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5deba9d3-2911-4fa6-9e4c-f61f023624ce None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.597813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de05d32-a4a0-48fa-b31a-3f475124d9d5 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.600579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8703d56e-976a-4ebf-a141-06e7f0e0695b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.601359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8703d56e-976a-4ebf-a141-06e7f0e0695b service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.601636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8703d56e-976a-4ebf-a141-06e7f0e0695b service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.601910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8703d56e-976a-4ebf-a141-06e7f0e0695b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.602831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597f41b6-2dcc-4b50-a809-c4112f3c300b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.603788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597f41b6-2dcc-4b50-a809-c4112f3c300b service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.604175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597f41b6-2dcc-4b50-a809-c4112f3c300b service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.604576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597f41b6-2dcc-4b50-a809-c4112f3c300b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.630225 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:35.659458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:35.689803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8703d56e-976a-4ebf-a141-06e7f0e0695b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.691433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8703d56e-976a-4ebf-a141-06e7f0e0695b service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.708385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 804/1619] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:35.708385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8703d56e-976a-4ebf-a141-06e7f0e0695b service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.718675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8703d56e-976a-4ebf-a141-06e7f0e0695b service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.725026 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628c023-6cd9-4250-8431-49b61a6f6bd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.725889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628c023-6cd9-4250-8431-49b61a6f6bd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.726236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628c023-6cd9-4250-8431-49b61a6f6bd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.726518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628c023-6cd9-4250-8431-49b61a6f6bd0 admin admin] PATH_INFO: `/v3/projects/5d640247639149d89c3742c29c9343c5/users/fa1735815cca4e388015297d369d1560/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.728732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597f41b6-2dcc-4b50-a809-c4112f3c300b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.733838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597f41b6-2dcc-4b50-a809-c4112f3c300b service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.746001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5628c023-6cd9-4250-8431-49b61a6f6bd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d640247639149d89c3742c29c9343c5, user_id=fa1735815cca4e388015297d369d1560, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.749327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5628c023-6cd9-4250-8431-49b61a6f6bd0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.760377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 816/1620] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:35.767273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807dc93-1847-4dbe-bacd-5edc7b0689be None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.767609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807dc93-1847-4dbe-bacd-5edc7b0689be None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.767912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7807dc93-1847-4dbe-bacd-5edc7b0689be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.773800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 805/1621] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:35 2022] PUT /identity/v3/projects/5d640247639149d89c3742c29c9343c5/users/fa1735815cca4e388015297d369d1560/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:35.773800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628c023-6cd9-4250-8431-49b61a6f6bd0 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.835359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7807dc93-1847-4dbe-bacd-5edc7b0689be None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:35.836725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7807dc93-1847-4dbe-bacd-5edc7b0689be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:35.858355 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628c023-6cd9-4250-8431-49b61a6f6bd0 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.864381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15562012-8e9b-4162-b98f-ce4d658af6f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.865236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15562012-8e9b-4162-b98f-ce4d658af6f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.866236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15562012-8e9b-4162-b98f-ce4d658af6f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.867212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15562012-8e9b-4162-b98f-ce4d658af6f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.868080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15562012-8e9b-4162-b98f-ce4d658af6f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.870399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15562012-8e9b-4162-b98f-ce4d658af6f6 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.903580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 806/1622] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/users => generated 300 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:35.905181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff829aa2-8efc-485b-87e5-2143866eea6c None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.905662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff829aa2-8efc-485b-87e5-2143866eea6c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.906155 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff829aa2-8efc-485b-87e5-2143866eea6c None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.908135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 807/1623] 10.222.0.27 () {54 vars in 962 bytes} [Thu Sep 15 14:09:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:09:35.909064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff829aa2-8efc-485b-87e5-2143866eea6c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.929165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff829aa2-8efc-485b-87e5-2143866eea6c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:35.939567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9273d47f-b91f-45cd-b85d-39ae44369152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:35.940635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9273d47f-b91f-45cd-b85d-39ae44369152 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.940920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9273d47f-b91f-45cd-b85d-39ae44369152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.941196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9273d47f-b91f-45cd-b85d-39ae44369152 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:35.941858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9273d47f-b91f-45cd-b85d-39ae44369152 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:35.943482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9273d47f-b91f-45cd-b85d-39ae44369152 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:35.969862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 808/1624] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:35 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:35.977160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 817/1625] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:35.979132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9273d47f-b91f-45cd-b85d-39ae44369152 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:35.980135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b886dda-1b88-4d59-aa47-5401d1b57339 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:35.980648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b886dda-1b88-4d59-aa47-5401d1b57339 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:35.981616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b886dda-1b88-4d59-aa47-5401d1b57339 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.004404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9273d47f-b91f-45cd-b85d-39ae44369152 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:36.018853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8b886dda-1b88-4d59-aa47-5401d1b57339 None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:36.019944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822d57d6-e307-4f1c-9fde-2d6557f1c84b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:36.020580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d57d6-e307-4f1c-9fde-2d6557f1c84b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.020815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8b886dda-1b88-4d59-aa47-5401d1b57339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:36.020950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d57d6-e307-4f1c-9fde-2d6557f1c84b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.021124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d57d6-e307-4f1c-9fde-2d6557f1c84b admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.021787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822d57d6-e307-4f1c-9fde-2d6557f1c84b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:36.023412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822d57d6-e307-4f1c-9fde-2d6557f1c84b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:36.038447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 818/1626] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Sep 15 14:09:35 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:36.039376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b886dda-1b88-4d59-aa47-5401d1b57339 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:36.049144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 809/1627] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:35 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:36.053227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822d57d6-e307-4f1c-9fde-2d6557f1c84b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:36.063389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b886dda-1b88-4d59-aa47-5401d1b57339 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:36.077955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ace2246-6b8f-42b7-910d-a4ab43843a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:36.079358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ace2246-6b8f-42b7-910d-a4ab43843a52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.079951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ace2246-6b8f-42b7-910d-a4ab43843a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.080234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ace2246-6b8f-42b7-910d-a4ab43843a52 admin admin] PATH_INFO: `/v3/projects/5d640247639149d89c3742c29c9343c5/users/fa1735815cca4e388015297d369d1560/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.083141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822d57d6-e307-4f1c-9fde-2d6557f1c84b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:36.087499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ace2246-6b8f-42b7-910d-a4ab43843a52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d640247639149d89c3742c29c9343c5, user_id=fa1735815cca4e388015297d369d1560, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:36.090365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ace2246-6b8f-42b7-910d-a4ab43843a52 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:36.100920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10faeec-62f6-4789-a6ab-9d179a12a05f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:36.101692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10faeec-62f6-4789-a6ab-9d179a12a05f service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.102034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10faeec-62f6-4789-a6ab-9d179a12a05f service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.102375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10faeec-62f6-4789-a6ab-9d179a12a05f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.113825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 819/1628] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:36 2022] PUT /identity/v3/projects/5d640247639149d89c3742c29c9343c5/users/fa1735815cca4e388015297d369d1560/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:36.121655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ace2246-6b8f-42b7-910d-a4ab43843a52 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:36.214626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10faeec-62f6-4789-a6ab-9d179a12a05f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:36.216591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10faeec-62f6-4789-a6ab-9d179a12a05f service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:36.229827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:36.241596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 810/1629] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:36 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:36.256427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10faeec-62f6-4789-a6ab-9d179a12a05f service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:36.278429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10faeec-62f6-4789-a6ab-9d179a12a05f service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:36.283094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ace2246-6b8f-42b7-910d-a4ab43843a52 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:36.296070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d76795d-fe8b-4918-aa0f-257732b5799a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:36.301800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d76795d-fe8b-4918-aa0f-257732b5799a admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.301800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d76795d-fe8b-4918-aa0f-257732b5799a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.301800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d76795d-fe8b-4918-aa0f-257732b5799a admin admin] PATH_INFO: `/v3/projects/c4be8dee593141b8a37653230e88817a/users/cab0c8f45a8e4866b013890d95dbd683/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.459115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d76795d-fe8b-4918-aa0f-257732b5799a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4be8dee593141b8a37653230e88817a, user_id=cab0c8f45a8e4866b013890d95dbd683, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:36.459115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d76795d-fe8b-4918-aa0f-257732b5799a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:36.459115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 820/1630] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:36 2022] PUT /identity/v3/projects/c4be8dee593141b8a37653230e88817a/users/cab0c8f45a8e4866b013890d95dbd683/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:36.459115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e8b1a-dce9-446d-a9cf-fd049a9ff9c4 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.459115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e8b1a-dce9-446d-a9cf-fd049a9ff9c4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.459115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e8b1a-dce9-446d-a9cf-fd049a9ff9c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.460759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc218cd-cccf-441b-b402-94f6814440e1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:36.461188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:36.465240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc218cd-cccf-441b-b402-94f6814440e1 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.465517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc218cd-cccf-441b-b402-94f6814440e1 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.465921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc218cd-cccf-441b-b402-94f6814440e1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.477074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9b6e8b1a-dce9-446d-a9cf-fd049a9ff9c4 None None] MFA Rules not processed for user `fa1735815cca4e388015297d369d1560`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:36.479516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9b6e8b1a-dce9-446d-a9cf-fd049a9ff9c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:36.610587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc218cd-cccf-441b-b402-94f6814440e1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:36.613772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc218cd-cccf-441b-b402-94f6814440e1 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:36.638705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 811/1631] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:36 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:36.640908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc218cd-cccf-441b-b402-94f6814440e1 service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:36.649393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 821/1632] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:36 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:36.659449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc218cd-cccf-441b-b402-94f6814440e1 service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:36.666673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6e8b1a-dce9-446d-a9cf-fd049a9ff9c4 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:36.778524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60367e00-7fc5-4a52-87a4-ad2f14cd9f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:36.779325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60367e00-7fc5-4a52-87a4-ad2f14cd9f4b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.779759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60367e00-7fc5-4a52-87a4-ad2f14cd9f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.780130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60367e00-7fc5-4a52-87a4-ad2f14cd9f4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.781195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60367e00-7fc5-4a52-87a4-ad2f14cd9f4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:36.783463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60367e00-7fc5-4a52-87a4-ad2f14cd9f4b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:36.797110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 812/1633] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:36 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:36.801885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60367e00-7fc5-4a52-87a4-ad2f14cd9f4b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:36.802063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6e8b1a-dce9-446d-a9cf-fd049a9ff9c4 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:36.816158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb31a1ac-b2d1-4e44-b4fc-c09d7402ccd7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:36.816793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31a1ac-b2d1-4e44-b4fc-c09d7402ccd7 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.817065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31a1ac-b2d1-4e44-b4fc-c09d7402ccd7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.817335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31a1ac-b2d1-4e44-b4fc-c09d7402ccd7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.818601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60367e00-7fc5-4a52-87a4-ad2f14cd9f4b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:36.824520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d49196-2f46-4732-b524-0724cff853a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:36.825761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d49196-2f46-4732-b524-0724cff853a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.826814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d49196-2f46-4732-b524-0724cff853a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.827338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d49196-2f46-4732-b524-0724cff853a3 admin admin] PATH_INFO: `/v3/projects/c4be8dee593141b8a37653230e88817a/users/cab0c8f45a8e4866b013890d95dbd683/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.830606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d49196-2f46-4732-b524-0724cff853a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4be8dee593141b8a37653230e88817a, user_id=cab0c8f45a8e4866b013890d95dbd683, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:36.833482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d49196-2f46-4732-b524-0724cff853a3 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:36.839748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb31a1ac-b2d1-4e44-b4fc-c09d7402ccd7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:36.841314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb31a1ac-b2d1-4e44-b4fc-c09d7402ccd7 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:36.849153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 813/1634] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:36 2022] PUT /identity/v3/projects/c4be8dee593141b8a37653230e88817a/users/cab0c8f45a8e4866b013890d95dbd683/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:36.860431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393a4624-21a3-4cac-84d4-7475a4c6781e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:36.860824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393a4624-21a3-4cac-84d4-7475a4c6781e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:36.861298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393a4624-21a3-4cac-84d4-7475a4c6781e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:36.866310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 822/1635] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:36 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:36.927814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-393a4624-21a3-4cac-84d4-7475a4c6781e None None] MFA Rules not processed for user `cab0c8f45a8e4866b013890d95dbd683`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:36.931933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-393a4624-21a3-4cac-84d4-7475a4c6781e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:36.961385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:37.139917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 814/1636] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:36 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:37.164888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb31a1ac-b2d1-4e44-b4fc-c09d7402ccd7 service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:37.178471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb31a1ac-b2d1-4e44-b4fc-c09d7402ccd7 service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:37.187138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91b808-695b-41af-8499-67309bafea54 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.187951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91b808-695b-41af-8499-67309bafea54 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.187951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c91b808-695b-41af-8499-67309bafea54 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:37.189300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 815/1637] 10.222.0.27 () {54 vars in 912 bytes} [Thu Sep 15 14:09:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:09:37.196198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785c2a46-c60c-43e7-b72a-b1a91576b85e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.196582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785c2a46-c60c-43e7-b72a-b1a91576b85e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.196800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785c2a46-c60c-43e7-b72a-b1a91576b85e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:37.233573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-785c2a46-c60c-43e7-b72a-b1a91576b85e None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:37.238622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-785c2a46-c60c-43e7-b72a-b1a91576b85e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:37.352022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a37560-9506-4d95-8734-b97ec5bef2e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:37.352628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a37560-9506-4d95-8734-b97ec5bef2e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.352929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a37560-9506-4d95-8734-b97ec5bef2e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.353267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a37560-9506-4d95-8734-b97ec5bef2e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:37.374468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a37560-9506-4d95-8734-b97ec5bef2e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:37.377280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a37560-9506-4d95-8734-b97ec5bef2e3 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:37.399348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 823/1638] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:37 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:09:37.417331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a37560-9506-4d95-8734-b97ec5bef2e3 service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:37.460987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 816/1639] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:09:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:37.462547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d51e61b-cd00-4eeb-8dc1-eabc27ec290b None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.462920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d51e61b-cd00-4eeb-8dc1-eabc27ec290b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.464257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d51e61b-cd00-4eeb-8dc1-eabc27ec290b None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:37.468636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 817/1640] 10.222.0.27 () {54 vars in 912 bytes} [Thu Sep 15 14:09:37 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:09:37.474336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbc3bf5-6a6a-4796-acbb-611f755e71dc None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.475061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbc3bf5-6a6a-4796-acbb-611f755e71dc None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.475357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbc3bf5-6a6a-4796-acbb-611f755e71dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:37.518031 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-1fbc3bf5-6a6a-4796-acbb-611f755e71dc None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:37.519359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1fbc3bf5-6a6a-4796-acbb-611f755e71dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:37.531790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 818/1641] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:09:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:37.532900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbc3bf5-6a6a-4796-acbb-611f755e71dc None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:37.542794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbc3bf5-6a6a-4796-acbb-611f755e71dc None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:37.552248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e98a21-1c71-4dbe-bb75-29bdbfe2c921 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:37.553066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e98a21-1c71-4dbe-bb75-29bdbfe2c921 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.553444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e98a21-1c71-4dbe-bb75-29bdbfe2c921 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.553820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e98a21-1c71-4dbe-bb75-29bdbfe2c921 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:37.583234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:37.596866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e98a21-1c71-4dbe-bb75-29bdbfe2c921 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:37.598488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e98a21-1c71-4dbe-bb75-29bdbfe2c921 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:37.614258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a37560-9506-4d95-8734-b97ec5bef2e3 service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:37.620218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:37.621621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.622067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.622356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] PATH_INFO: `/v3/users/09e7e3f3c2a84a1aa1e16324312a57f6` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:37.623553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] RBAC: Authorizing `identity:delete_user(user_id=09e7e3f3c2a84a1aa1e16324312a57f6)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:37.627751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 819/1642] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:37.628913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e98a21-1c71-4dbe-bb75-29bdbfe2c921 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:37.649183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: WARNING py.warnings [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:09:37.649183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: warnings.warn(msg) Sep 15 14:09:37.649183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: Sep 15 14:09:37.652775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:37.653444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e98a21-1c71-4dbe-bb75-29bdbfe2c921 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:37.668866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:37.669504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.669886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.670258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] PATH_INFO: `/v3/users/90d193d31f0c43999fdb052d28e7d833` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:37.672019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=90d193d31f0c43999fdb052d28e7d833)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:37.675179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:37.780674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90d193d31f0c43999fdb052d28e7d833'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.798476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90d193d31f0c43999fdb052d28e7d833'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.799289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90d193d31f0c43999fdb052d28e7d833'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.800079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90d193d31f0c43999fdb052d28e7d833'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.814115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90d193d31f0c43999fdb052d28e7d833'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.830661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:37.839968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09e7e3f3c2a84a1aa1e16324312a57f6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.851748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 820/1643] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:09:37 2022] DELETE /identity/v3/users/90d193d31f0c43999fdb052d28e7d833 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:37.852798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:37.864199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:37.873457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09e7e3f3c2a84a1aa1e16324312a57f6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.874407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09e7e3f3c2a84a1aa1e16324312a57f6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.876011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09e7e3f3c2a84a1aa1e16324312a57f6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.907630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6da619e3-f964-4c3c-8701-2af156148505 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09e7e3f3c2a84a1aa1e16324312a57f6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:37.973668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 824/1644] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:09:37 2022] DELETE /identity/v3/users/09e7e3f3c2a84a1aa1e16324312a57f6 => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:37.981778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077f28e-34b6-41da-a7db-7182c6b74266 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:37.981778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077f28e-34b6-41da-a7db-7182c6b74266 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:37.981778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077f28e-34b6-41da-a7db-7182c6b74266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:38.018149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8501a43-bb6f-4363-8b00-442a98ba0d45 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:38.035032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1077f28e-34b6-41da-a7db-7182c6b74266 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:38.037407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1077f28e-34b6-41da-a7db-7182c6b74266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.126772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:38.127912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:38.128380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:38.128842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:38.132102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.162640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 825/1645] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:38.164437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5487de-b191-437f-abbb-f4253ac589ee None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:38.164954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5487de-b191-437f-abbb-f4253ac589ee None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:38.165432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5487de-b191-437f-abbb-f4253ac589ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:38.188751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-db5487de-b191-437f-abbb-f4253ac589ee None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:38.190448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-db5487de-b191-437f-abbb-f4253ac589ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.200606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 826/1646] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:38.201764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5487de-b191-437f-abbb-f4253ac589ee None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:38.203584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-db5487de-b191-437f-abbb-f4253ac589ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.216625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5487de-b191-437f-abbb-f4253ac589ee None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:38.223992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:38.224978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:38.225490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:38.225971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] PATH_INFO: `/v3/users/d9baa62cc65147719248fe7e898dbb8d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:38.227419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9baa62cc65147719248fe7e898dbb8d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:38.229824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:38.240899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:38.242445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:38.268166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 821/1647] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:38.270328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:38.274974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.328684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9baa62cc65147719248fe7e898dbb8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:38.335890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:38.355442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9baa62cc65147719248fe7e898dbb8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:38.356337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9baa62cc65147719248fe7e898dbb8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:38.356988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9baa62cc65147719248fe7e898dbb8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:38.379128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9baa62cc65147719248fe7e898dbb8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:38.419856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 827/1648] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:09:38 2022] DELETE /identity/v3/users/d9baa62cc65147719248fe7e898dbb8d => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:38.419856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:38.419856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.433567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:38.447458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e6a58-54fe-4d90-9f85-43c92ec524b9 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:38.541330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f27b940-b5c1-490f-a537-ff52e3fe3908 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:38.596798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f303f59-7b69-488a-9c3c-b312d17ad383 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:38.596798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f303f59-7b69-488a-9c3c-b312d17ad383 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:38.596798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f303f59-7b69-488a-9c3c-b312d17ad383 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:38.596798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f303f59-7b69-488a-9c3c-b312d17ad383 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:38.601176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6f303f59-7b69-488a-9c3c-b312d17ad383 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.616011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:38.618341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:38.618758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:38.620495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:38.626158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.735463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f303f59-7b69-488a-9c3c-b312d17ad383 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:38.740414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f303f59-7b69-488a-9c3c-b312d17ad383 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:38.770364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 828/1649] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:38.772668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edae4bf-5123-4154-b08a-1e87bc0baf66 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:38.773093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edae4bf-5123-4154-b08a-1e87bc0baf66 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:38.774638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edae4bf-5123-4154-b08a-1e87bc0baf66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:38.783894 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:38.785543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:38.806608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 822/1650] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:38.829512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5edae4bf-5123-4154-b08a-1e87bc0baf66 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:38.838939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5edae4bf-5123-4154-b08a-1e87bc0baf66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:38.855665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 829/1651] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:38.984219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:38.986551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:39.006892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15efe2d8-23e0-4521-b7d0-554d720b959c service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:39.019506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577bc78-0e2f-4c3e-8862-ed6b6a2ce29c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:39.020359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577bc78-0e2f-4c3e-8862-ed6b6a2ce29c service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:39.020645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577bc78-0e2f-4c3e-8862-ed6b6a2ce29c service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:39.020981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577bc78-0e2f-4c3e-8862-ed6b6a2ce29c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:39.039581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2577bc78-0e2f-4c3e-8862-ed6b6a2ce29c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:39.041082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2577bc78-0e2f-4c3e-8862-ed6b6a2ce29c service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:39.062493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 823/1652] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:09:40.702381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edae4bf-5123-4154-b08a-1e87bc0baf66 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:40.703138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5edae4bf-5123-4154-b08a-1e87bc0baf66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:40.731224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edae4bf-5123-4154-b08a-1e87bc0baf66 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:40.747546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:40.748247 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:40.748523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:40.748794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] PATH_INFO: `/v3/projects/a84b43c2ba1f4497858e2be943acf2d4` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:40.751405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a84b43c2ba1f4497858e2be943acf2d4)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:40.753789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:40.836539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a84b43c2ba1f4497858e2be943acf2d4', 'request_id': 'req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:40.854726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a84b43c2ba1f4497858e2be943acf2d4', 'request_id': 'req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:40.863044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a84b43c2ba1f4497858e2be943acf2d4', 'request_id': 'req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:40.864570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:40.892037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] The token cache is being invalidate because project a84b43c2ba1f4497858e2be943acf2d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:40.892676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:40.894701 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:40.898317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 830/1653] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:40 2022] DELETE /identity/v3/projects/a84b43c2ba1f4497858e2be943acf2d4 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:40.901912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577bc78-0e2f-4c3e-8862-ed6b6a2ce29c service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:40.920317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:40.923154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:40.928948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577bc78-0e2f-4c3e-8862-ed6b6a2ce29c service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:41.196283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:41.197215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:41.197648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:41.197932 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] PATH_INFO: `/v3/users/fa1735815cca4e388015297d369d1560` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:41.202764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fa1735815cca4e388015297d369d1560)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:41.208383 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:41.220603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6051a2e-0ec6-4790-b81d-f6bd3bf69202 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:41.254427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:41.255487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:41.255487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:41.255665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] PATH_INFO: `/v3/projects/96b37cc8c62e4096b072abff9b051def` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:41.259062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] RBAC: Authorizing `identity:delete_project(project_id=96b37cc8c62e4096b072abff9b051def)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:41.261308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:41.358746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:41.369938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fa1735815cca4e388015297d369d1560'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.383419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96b37cc8c62e4096b072abff9b051def', 'request_id': 'req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.393418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fa1735815cca4e388015297d369d1560'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.394199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fa1735815cca4e388015297d369d1560'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.395779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fa1735815cca4e388015297d369d1560'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.402125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96b37cc8c62e4096b072abff9b051def', 'request_id': 'req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.420076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fa1735815cca4e388015297d369d1560'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.422865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96b37cc8c62e4096b072abff9b051def', 'request_id': 'req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.449604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 824/1654] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:09:40 2022] DELETE /identity/v3/users/fa1735815cca4e388015297d369d1560 => generated 0 bytes in 548 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:41.450545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:41.466406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:41.487007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] The token cache is being invalidate because project 96b37cc8c62e4096b072abff9b051def was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:41.487977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.494416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a2809d4c-8f56-454e-9ca5-d99e9c7e198b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.497263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 831/1655] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:40 2022] DELETE /identity/v3/projects/96b37cc8c62e4096b072abff9b051def => generated 0 bytes in 578 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:41.502474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1838e561-4f84-46d0-b8cb-154fa44002ae None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:41.502753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1838e561-4f84-46d0-b8cb-154fa44002ae None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:41.503026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1838e561-4f84-46d0-b8cb-154fa44002ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:41.559649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1838e561-4f84-46d0-b8cb-154fa44002ae None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:41.561017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1838e561-4f84-46d0-b8cb-154fa44002ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:41.566152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8a1d42-4968-4f98-97ac-7e8aef0fd9d3 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:41.571148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:41.571837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:41.572161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:41.572436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] PATH_INFO: `/v3/projects/3f927dff41404c4ba81b5d8882c99dd4` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:41.573791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f927dff41404c4ba81b5d8882c99dd4)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:41.575116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: WARNING py.warnings [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 15 14:09:41.575116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: warnings.warn(msg) Sep 15 14:09:41.575116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: Sep 15 14:09:41.576544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:41.584796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 832/1656] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:41.593596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245dfbed-07ad-4f6e-b343-36be2280fc2d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:41.593874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245dfbed-07ad-4f6e-b343-36be2280fc2d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:41.594782 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245dfbed-07ad-4f6e-b343-36be2280fc2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:41.644069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-245dfbed-07ad-4f6e-b343-36be2280fc2d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:41.645320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-245dfbed-07ad-4f6e-b343-36be2280fc2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:41.663066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f927dff41404c4ba81b5d8882c99dd4', 'request_id': 'req-62ae5d9d-ebd7-4789-a54b-5edad05cc817'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.676067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f927dff41404c4ba81b5d8882c99dd4', 'request_id': 'req-62ae5d9d-ebd7-4789-a54b-5edad05cc817'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.687290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f927dff41404c4ba81b5d8882c99dd4', 'request_id': 'req-62ae5d9d-ebd7-4789-a54b-5edad05cc817'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.721082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] The token cache is being invalidate because project 3f927dff41404c4ba81b5d8882c99dd4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:41.721632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.722552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:41.725389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 825/1657] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:41 2022] DELETE /identity/v3/projects/3f927dff41404c4ba81b5d8882c99dd4 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:41.732233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:41.735552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:41.795262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 833/1658] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:41.800751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245dfbed-07ad-4f6e-b343-36be2280fc2d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:41.814281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245dfbed-07ad-4f6e-b343-36be2280fc2d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:41.834233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:41.847170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:41.848556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:41.849001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:41.849422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:41.850955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:41.851868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:41.854900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:41.858710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae5d9d-ebd7-4789-a54b-5edad05cc817 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:41.866708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:41.868603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:41.869584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:41.870084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] PATH_INFO: `/v3/users/cab0c8f45a8e4866b013890d95dbd683` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:41.871991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cab0c8f45a8e4866b013890d95dbd683)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:41.880385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 834/1659] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:41.887906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:41.888822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:41.928860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:41.938314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09210aa-cf39-4d16-99d1-59f047c11c12 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:41.954730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:41.955491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:41.955874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:41.957295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:41.958723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:41.964447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:42.025938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cab0c8f45a8e4866b013890d95dbd683'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:42.036786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cab0c8f45a8e4866b013890d95dbd683'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:42.037884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cab0c8f45a8e4866b013890d95dbd683'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:42.038876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cab0c8f45a8e4866b013890d95dbd683'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:42.048639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 835/1660] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:41 2022] POST /identity/v3/projects => generated 371 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:42.049723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:42.054801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:42.060712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cab0c8f45a8e4866b013890d95dbd683'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:42.090020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 826/1661] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:09:41 2022] DELETE /identity/v3/users/cab0c8f45a8e4866b013890d95dbd683 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:42.215391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b605b03-a4d6-4dc1-b309-28407447099f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:42.419829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475fd9c6-d942-4517-9687-9e7fe77bd167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:42.419829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475fd9c6-d942-4517-9687-9e7fe77bd167 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:42.419829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475fd9c6-d942-4517-9687-9e7fe77bd167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:42.419829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475fd9c6-d942-4517-9687-9e7fe77bd167 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:42.420194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475fd9c6-d942-4517-9687-9e7fe77bd167 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:42.420194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475fd9c6-d942-4517-9687-9e7fe77bd167 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:42.474544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 836/1662] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:42 2022] POST /identity/v3/users => generated 286 bytes in 422 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:42.476149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:42.499151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ac32f-5ffb-4761-abc1-fcdfd2bec373 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:42.517443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e959dc7-339b-48ef-9a04-74a0cf1ab4e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:42.518333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e959dc7-339b-48ef-9a04-74a0cf1ab4e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:42.518720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e959dc7-339b-48ef-9a04-74a0cf1ab4e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:42.519095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e959dc7-339b-48ef-9a04-74a0cf1ab4e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:42.520822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e959dc7-339b-48ef-9a04-74a0cf1ab4e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:42.526055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e959dc7-339b-48ef-9a04-74a0cf1ab4e5 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:42.556222 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475fd9c6-d942-4517-9687-9e7fe77bd167 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:42.569474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 827/1663] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:42 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:42.575932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475fd9c6-d942-4517-9687-9e7fe77bd167 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:42.588886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8bb3f2-e5c8-4e75-b939-2b67139079b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:42.590778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8bb3f2-e5c8-4e75-b939-2b67139079b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:42.591234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8bb3f2-e5c8-4e75-b939-2b67139079b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:42.593102 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8bb3f2-e5c8-4e75-b939-2b67139079b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:42.599609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8bb3f2-e5c8-4e75-b939-2b67139079b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:42.601559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8bb3f2-e5c8-4e75-b939-2b67139079b5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:42.630238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 837/1664] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:42 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:42.632128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e959dc7-339b-48ef-9a04-74a0cf1ab4e5 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:42.644352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e959dc7-339b-48ef-9a04-74a0cf1ab4e5 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:42.674934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:42.676053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:42.676569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:42.677038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] PATH_INFO: `/v3/projects/03facb46dcfc4545a87e5980260048ea/users/48083c132cf94804a2ab5732f1e3e825/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:42.724492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03facb46dcfc4545a87e5980260048ea, user_id=48083c132cf94804a2ab5732f1e3e825, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:42.732115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:42.767088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 828/1665] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:42 2022] PUT /identity/v3/projects/03facb46dcfc4545a87e5980260048ea/users/48083c132cf94804a2ab5732f1e3e825/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:09:42.778911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b98880-72a2-40c6-935d-47e5f1a671ea None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:42.779367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b98880-72a2-40c6-935d-47e5f1a671ea None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:42.780159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b98880-72a2-40c6-935d-47e5f1a671ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:42.790150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:42.792287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:42.904355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-99b98880-72a2-40c6-935d-47e5f1a671ea None None] MFA Rules not processed for user `48083c132cf94804a2ab5732f1e3e825`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:42.905614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-99b98880-72a2-40c6-935d-47e5f1a671ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:42.987908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8ac1c4-62b1-4d63-a22b-521a080fe088 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:43.001020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:43.002611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:43.002611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:43.003740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] PATH_INFO: `/v3/projects/5d640247639149d89c3742c29c9343c5` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:43.005187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d640247639149d89c3742c29c9343c5)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:43.018684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:43.060612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:43.072248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 838/1666] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:42 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:43.076295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b98880-72a2-40c6-935d-47e5f1a671ea None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:43.113016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b98880-72a2-40c6-935d-47e5f1a671ea None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:43.126868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7c2eac-4d93-44fc-b679-94b7637b72d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:43.127227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c2eac-4d93-44fc-b679-94b7637b72d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:43.127507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c2eac-4d93-44fc-b679-94b7637b72d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:43.128137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c2eac-4d93-44fc-b679-94b7637b72d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:43.128803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7c2eac-4d93-44fc-b679-94b7637b72d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:43.130734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7c2eac-4d93-44fc-b679-94b7637b72d9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:43.187568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:43.238839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 839/1667] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:43 2022] POST /identity/v3/projects => generated 373 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:09:43.239305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d640247639149d89c3742c29c9343c5', 'request_id': 'req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.245175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7c2eac-4d93-44fc-b679-94b7637b72d9 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:43.269479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d640247639149d89c3742c29c9343c5', 'request_id': 'req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.288029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d640247639149d89c3742c29c9343c5', 'request_id': 'req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.342218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] The token cache is being invalidate because project 5d640247639149d89c3742c29c9343c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:43.342585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.344033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.368348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 829/1668] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:42 2022] DELETE /identity/v3/projects/5d640247639149d89c3742c29c9343c5 => generated 0 bytes in 575 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:43.369450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:43.370904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:43.400371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7c2eac-4d93-44fc-b679-94b7637b72d9 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:43.420968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0274dff8-273f-4fd5-8297-fb47d8a7283b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:43.423267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274dff8-273f-4fd5-8297-fb47d8a7283b admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:43.423638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274dff8-273f-4fd5-8297-fb47d8a7283b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:43.424068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274dff8-273f-4fd5-8297-fb47d8a7283b admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:43.424904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0274dff8-273f-4fd5-8297-fb47d8a7283b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:43.429625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0274dff8-273f-4fd5-8297-fb47d8a7283b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:43.505834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 840/1669] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:43 2022] POST /identity/v3/users => generated 293 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:43.507347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc68c17c-a911-4246-b462-50db81b82c8f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:43.507631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc68c17c-a911-4246-b462-50db81b82c8f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:43.507938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc68c17c-a911-4246-b462-50db81b82c8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:43.511076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa135c2-afc6-4f5f-81fe-5bb67e7f0d96 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:43.530107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:43.530788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:43.531252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:43.531771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] PATH_INFO: `/v3/projects/c4be8dee593141b8a37653230e88817a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:43.533860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4be8dee593141b8a37653230e88817a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:43.535872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:43.577278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-dc68c17c-a911-4246-b462-50db81b82c8f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:43.583551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dc68c17c-a911-4246-b462-50db81b82c8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:43.613423 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 841/1670] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:43.614295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc68c17c-a911-4246-b462-50db81b82c8f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:43.618892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dc68c17c-a911-4246-b462-50db81b82c8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:43.660757 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc68c17c-a911-4246-b462-50db81b82c8f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:43.671987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4be8dee593141b8a37653230e88817a', 'request_id': 'req-2400a38a-79c5-47fe-87e5-6f85c37122af'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.691301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4be8dee593141b8a37653230e88817a', 'request_id': 'req-2400a38a-79c5-47fe-87e5-6f85c37122af'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.698775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4be8dee593141b8a37653230e88817a', 'request_id': 'req-2400a38a-79c5-47fe-87e5-6f85c37122af'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.724603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] The token cache is being invalidate because project c4be8dee593141b8a37653230e88817a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:43.724998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.728907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:43.735929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 830/1671] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:43 2022] DELETE /identity/v3/projects/c4be8dee593141b8a37653230e88817a => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:43.736940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:43.738989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:43.888559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:43.933960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2400a38a-79c5-47fe-87e5-6f85c37122af admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:43.943117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:43.944544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:43.945496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:43.945979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] PATH_INFO: `/v3/users/22bd431202dc4bb7a3e2741a7c1a51b8` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:43.947407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] RBAC: Authorizing `identity:delete_user(user_id=22bd431202dc4bb7a3e2741a7c1a51b8)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:43.949609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:43.973936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91bd535-3b3a-4643-bb09-773716963851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:43.974758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91bd535-3b3a-4643-bb09-773716963851 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:43.978056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91bd535-3b3a-4643-bb09-773716963851 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:43.978483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91bd535-3b3a-4643-bb09-773716963851 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:43.979272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91bd535-3b3a-4643-bb09-773716963851 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:43.982168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91bd535-3b3a-4643-bb09-773716963851 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:44.019032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 831/1672] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:43 2022] GET /identity/v3/roles => generated 1420 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:44.019032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fce1f-06e2-43e9-aa47-f6d19354e455 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.019032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fce1f-06e2-43e9-aa47-f6d19354e455 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.019032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fce1f-06e2-43e9-aa47-f6d19354e455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.061835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-390fce1f-06e2-43e9-aa47-f6d19354e455 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:44.064005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-390fce1f-06e2-43e9-aa47-f6d19354e455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.076187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '22bd431202dc4bb7a3e2741a7c1a51b8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:44.089375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 832/1673] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:44.089375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98e25c-53c8-4853-90e6-c9fa12ae3d9e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.089375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98e25c-53c8-4853-90e6-c9fa12ae3d9e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.089375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98e25c-53c8-4853-90e6-c9fa12ae3d9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.098063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '22bd431202dc4bb7a3e2741a7c1a51b8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:44.098835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '22bd431202dc4bb7a3e2741a7c1a51b8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:44.100357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '22bd431202dc4bb7a3e2741a7c1a51b8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:44.115036 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=87,keystone:INSERT=5,keystone:DELETE=8 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:44.119050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-455696f7-c778-457e-960a-76ad0be2ca7a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '22bd431202dc4bb7a3e2741a7c1a51b8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:44.159008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4d98e25c-53c8-4853-90e6-c9fa12ae3d9e None None] MFA Rules not processed for user `2049d66708ce47a794c14ad4c2040937`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:44.160627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4d98e25c-53c8-4853-90e6-c9fa12ae3d9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.165730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 842/1674] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:09:43 2022] DELETE /identity/v3/users/22bd431202dc4bb7a3e2741a7c1a51b8 => generated 0 bytes in 552 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:44.167138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e6fb6-b7ed-4098-bd68-3c5101980a17 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.167427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e6fb6-b7ed-4098-bd68-3c5101980a17 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.167728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e6fb6-b7ed-4098-bd68-3c5101980a17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.202217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-523e6fb6-b7ed-4098-bd68-3c5101980a17 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:44.203597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-523e6fb6-b7ed-4098-bd68-3c5101980a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.298241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 833/1675] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:44 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:44.304549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d98e25c-53c8-4853-90e6-c9fa12ae3d9e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:44.318804 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4d98e25c-53c8-4853-90e6-c9fa12ae3d9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.341436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d98e25c-53c8-4853-90e6-c9fa12ae3d9e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:44.387045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 843/1676] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:44.396244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523e6fb6-b7ed-4098-bd68-3c5101980a17 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:44.398254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-523e6fb6-b7ed-4098-bd68-3c5101980a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.413223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523e6fb6-b7ed-4098-bd68-3c5101980a17 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:44.419241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5321e5-c309-4d42-a5f1-e520d28de006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:44.420318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5321e5-c309-4d42-a5f1-e520d28de006 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.420685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5321e5-c309-4d42-a5f1-e520d28de006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.421014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5321e5-c309-4d42-a5f1-e520d28de006 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.421748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5321e5-c309-4d42-a5f1-e520d28de006 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:44.422723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5321e5-c309-4d42-a5f1-e520d28de006 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:44.427969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5321e5-c309-4d42-a5f1-e520d28de006 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:44.447782 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 844/1677] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:44.448505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:44.449246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.449526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.449866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] PATH_INFO: `/v3/projects/1496d6f83bb64c2b9013f2695d24e426/users/ce6a69b266c64952b94540f3a70e244f/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.450736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466a7edb-c118-4256-825e-a1e1dc9d08a6 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.451354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466a7edb-c118-4256-825e-a1e1dc9d08a6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.451837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466a7edb-c118-4256-825e-a1e1dc9d08a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.485261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1496d6f83bb64c2b9013f2695d24e426, user_id=ce6a69b266c64952b94540f3a70e244f, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:44.489075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:44.508178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-466a7edb-c118-4256-825e-a1e1dc9d08a6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:44.511280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-466a7edb-c118-4256-825e-a1e1dc9d08a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.519037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 834/1678] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:44 2022] PUT /identity/v3/projects/1496d6f83bb64c2b9013f2695d24e426/users/ce6a69b266c64952b94540f3a70e244f/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:44.522814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:44.524630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.623638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 845/1679] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:44.629929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bba724-9065-4052-b2ec-297060e68d1e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.630444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bba724-9065-4052-b2ec-297060e68d1e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.631204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bba724-9065-4052-b2ec-297060e68d1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.642449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:44.669030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-45bba724-9065-4052-b2ec-297060e68d1e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:44.670326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-45bba724-9065-4052-b2ec-297060e68d1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.685316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 846/1680] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:44.686299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bba724-9065-4052-b2ec-297060e68d1e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:44.690350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e6ec3a-c132-494e-b37e-d4457aef0090 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:44.701903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bba724-9065-4052-b2ec-297060e68d1e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:44.702724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:44.704850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.705367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.705964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.711735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eff8d61-fde6-4b20-a24a-87d810b72afa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:44.712361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eff8d61-fde6-4b20-a24a-87d810b72afa admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.712643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eff8d61-fde6-4b20-a24a-87d810b72afa admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.712917 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eff8d61-fde6-4b20-a24a-87d810b72afa admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.716577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eff8d61-fde6-4b20-a24a-87d810b72afa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:44.724210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eff8d61-fde6-4b20-a24a-87d810b72afa admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:44.785106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 847/1681] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:44 2022] POST /identity/v3/projects => generated 381 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:44.788200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eff8d61-fde6-4b20-a24a-87d810b72afa admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:44.836071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:44.845127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:44.867351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 835/1682] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:44 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:44.871077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:44.872889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:44.892251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:44.948699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eff8d61-fde6-4b20-a24a-87d810b72afa admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:44.963777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a19c194-04e0-49b9-9dcd-04ac945f1f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:44.964890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a19c194-04e0-49b9-9dcd-04ac945f1f6c admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:44.966235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a19c194-04e0-49b9-9dcd-04ac945f1f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:44.966235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a19c194-04e0-49b9-9dcd-04ac945f1f6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:44.966235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a19c194-04e0-49b9-9dcd-04ac945f1f6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:44.968100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a19c194-04e0-49b9-9dcd-04ac945f1f6c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:44.987890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 848/1683] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:44 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:44.988785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a19c194-04e0-49b9-9dcd-04ac945f1f6c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.002211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27f4e2b-910d-4176-8497-f83347cbbc31 service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.013878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a19c194-04e0-49b9-9dcd-04ac945f1f6c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.029274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.029916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.030199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.030471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.031139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:45.031489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.035677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.065637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 849/1684] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:45.065975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.102764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43094dd7-8c4e-492e-ae6c-88482e213abb admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.119384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950a9f0e-34c5-4586-919e-fbee082d5439 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.120311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950a9f0e-34c5-4586-919e-fbee082d5439 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.120732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950a9f0e-34c5-4586-919e-fbee082d5439 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.121182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950a9f0e-34c5-4586-919e-fbee082d5439 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.121592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d85dfa-4bc9-47a0-8031-c58421130d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.122262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d85dfa-4bc9-47a0-8031-c58421130d75 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.122543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d85dfa-4bc9-47a0-8031-c58421130d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.122820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d85dfa-4bc9-47a0-8031-c58421130d75 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.125843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d85dfa-4bc9-47a0-8031-c58421130d75 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.140167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d85dfa-4bc9-47a0-8031-c58421130d75 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.150001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950a9f0e-34c5-4586-919e-fbee082d5439 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.151950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950a9f0e-34c5-4586-919e-fbee082d5439 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.178517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 836/1685] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:45.179464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950a9f0e-34c5-4586-919e-fbee082d5439 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.200931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950a9f0e-34c5-4586-919e-fbee082d5439 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.206135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 850/1686] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:45 2022] POST /identity/v3/users => generated 291 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:45.207040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d85dfa-4bc9-47a0-8031-c58421130d75 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.210712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049c5221-3708-4c19-ac2f-d4a370261941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.211165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049c5221-3708-4c19-ac2f-d4a370261941 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.211544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049c5221-3708-4c19-ac2f-d4a370261941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.211944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049c5221-3708-4c19-ac2f-d4a370261941 admin admin] PATH_INFO: `/v3/projects/1496d6f83bb64c2b9013f2695d24e426/users/ce6a69b266c64952b94540f3a70e244f/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.214710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049c5221-3708-4c19-ac2f-d4a370261941 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1496d6f83bb64c2b9013f2695d24e426, user_id=ce6a69b266c64952b94540f3a70e244f, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.217830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049c5221-3708-4c19-ac2f-d4a370261941 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.222405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d85dfa-4bc9-47a0-8031-c58421130d75 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.229389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8dbd24-a350-49dd-a102-20a589339ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.230016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8dbd24-a350-49dd-a102-20a589339ea7 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.230293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8dbd24-a350-49dd-a102-20a589339ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.231154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8dbd24-a350-49dd-a102-20a589339ea7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.231848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8dbd24-a350-49dd-a102-20a589339ea7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.235963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8dbd24-a350-49dd-a102-20a589339ea7 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.242407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 837/1687] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:45 2022] PUT /identity/v3/projects/1496d6f83bb64c2b9013f2695d24e426/users/ce6a69b266c64952b94540f3a70e244f/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:45.243600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049c5221-3708-4c19-ac2f-d4a370261941 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.276920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 851/1688] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:45 2022] POST /identity/v3/projects => generated 391 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:45.278413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8897fe62-9f51-437f-bf8d-42351804de7f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.279161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8897fe62-9f51-437f-bf8d-42351804de7f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.279161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8897fe62-9f51-437f-bf8d-42351804de7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.314977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:45.336327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8897fe62-9f51-437f-bf8d-42351804de7f None None] MFA Rules not processed for user `ce6a69b266c64952b94540f3a70e244f`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:45.337704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8897fe62-9f51-437f-bf8d-42351804de7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:45.346719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049c5221-3708-4c19-ac2f-d4a370261941 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.416805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68454399-edea-4a5f-97ce-d9067536fc14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.417433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68454399-edea-4a5f-97ce-d9067536fc14 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.417717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68454399-edea-4a5f-97ce-d9067536fc14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.418009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68454399-edea-4a5f-97ce-d9067536fc14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.418798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68454399-edea-4a5f-97ce-d9067536fc14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.420469 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68454399-edea-4a5f-97ce-d9067536fc14 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.436794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 838/1689] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:45 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:45.436794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68454399-edea-4a5f-97ce-d9067536fc14 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.444202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 852/1690] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:45 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:45.445540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8897fe62-9f51-437f-bf8d-42351804de7f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.445775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68454399-edea-4a5f-97ce-d9067536fc14 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.452087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f9dbcf-556a-435c-b677-fed0c6dcf5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.452710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f9dbcf-556a-435c-b677-fed0c6dcf5e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.452992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f9dbcf-556a-435c-b677-fed0c6dcf5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.453787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f9dbcf-556a-435c-b677-fed0c6dcf5e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.454613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f9dbcf-556a-435c-b677-fed0c6dcf5e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.460369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f9dbcf-556a-435c-b677-fed0c6dcf5e7 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.463425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8897fe62-9f51-437f-bf8d-42351804de7f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.472065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.472939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.473375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.473789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.474587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.476785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.499437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 853/1691] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:45 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:45.502966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.505192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:45.508002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 839/1692] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:45 2022] POST /identity/v3/users => generated 296 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:45.508970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f9dbcf-556a-435c-b677-fed0c6dcf5e7 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.536457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f9dbcf-556a-435c-b677-fed0c6dcf5e7 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.545832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8a779-f853-4832-b0f5-aaac9c4f3050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.547372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8a779-f853-4832-b0f5-aaac9c4f3050 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.548580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8a779-f853-4832-b0f5-aaac9c4f3050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.549553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8a779-f853-4832-b0f5-aaac9c4f3050 admin admin] PATH_INFO: `/v3/projects/1c79952eb7644ad2842f103ee5ff9e20/users/c400909b5c434bfbb3686107b570dc26/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.590756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b8a779-f853-4832-b0f5-aaac9c4f3050 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c79952eb7644ad2842f103ee5ff9e20, user_id=c400909b5c434bfbb3686107b570dc26, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.599917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b8a779-f853-4832-b0f5-aaac9c4f3050 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.638987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 840/1693] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:45 2022] PUT /identity/v3/projects/1c79952eb7644ad2842f103ee5ff9e20/users/c400909b5c434bfbb3686107b570dc26/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:45.641731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8a779-f853-4832-b0f5-aaac9c4f3050 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.661914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f914678e-a8e8-47ac-a428-d61afa42a718 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.686206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:45.783496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc0043b-eb13-47c6-be62-7719355bb06e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.784810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc0043b-eb13-47c6-be62-7719355bb06e service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.785400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc0043b-eb13-47c6-be62-7719355bb06e service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.786455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8a779-f853-4832-b0f5-aaac9c4f3050 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.786877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc0043b-eb13-47c6-be62-7719355bb06e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.793177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2b2f25-ae80-4418-9a08-89401eee2925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.793965 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2b2f25-ae80-4418-9a08-89401eee2925 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.794249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2b2f25-ae80-4418-9a08-89401eee2925 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.794571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2b2f25-ae80-4418-9a08-89401eee2925 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.795409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2b2f25-ae80-4418-9a08-89401eee2925 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.797364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2b2f25-ae80-4418-9a08-89401eee2925 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.811140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 841/1694] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:45 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:45.815094 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd05b54-df3f-4a88-966a-aeb456a1f28c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.815802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd05b54-df3f-4a88-966a-aeb456a1f28c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.816274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd05b54-df3f-4a88-966a-aeb456a1f28c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.850399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ebd05b54-df3f-4a88-966a-aeb456a1f28c None None] MFA Rules not processed for user `c400909b5c434bfbb3686107b570dc26`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:45.851627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ebd05b54-df3f-4a88-966a-aeb456a1f28c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:45.885682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc0043b-eb13-47c6-be62-7719355bb06e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.888752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc0043b-eb13-47c6-be62-7719355bb06e service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.897044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:45.902840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 854/1695] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:45 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:45.903879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc0043b-eb13-47c6-be62-7719355bb06e service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.917252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc0043b-eb13-47c6-be62-7719355bb06e service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.923846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666eb54-2439-4bdc-b06e-55aa4dbca1ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.924586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666eb54-2439-4bdc-b06e-55aa4dbca1ee admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.924966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666eb54-2439-4bdc-b06e-55aa4dbca1ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.925337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666eb54-2439-4bdc-b06e-55aa4dbca1ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.926099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f666eb54-2439-4bdc-b06e-55aa4dbca1ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.927835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f666eb54-2439-4bdc-b06e-55aa4dbca1ee admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:45.940615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 855/1696] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:45 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:45.945048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666eb54-2439-4bdc-b06e-55aa4dbca1ee admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.957876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666eb54-2439-4bdc-b06e-55aa4dbca1ee admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.964152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f965e95a-ad76-4e04-af04-bd6d1c867475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:45.965188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965e95a-ad76-4e04-af04-bd6d1c867475 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:45.965516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965e95a-ad76-4e04-af04-bd6d1c867475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:45.965888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965e95a-ad76-4e04-af04-bd6d1c867475 admin admin] PATH_INFO: `/v3/projects/787051300d714b628ecbe666880d0854/users/cdf9054534b6474f88f216794ff39593/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:45.973888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 842/1697] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:45 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:45.974812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd05b54-df3f-4a88-966a-aeb456a1f28c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:45.987231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f965e95a-ad76-4e04-af04-bd6d1c867475 admin admin] RBAC: Authorizing `identity:create_grant(project_id=787051300d714b628ecbe666880d0854, user_id=cdf9054534b6474f88f216794ff39593, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:45.990140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd05b54-df3f-4a88-966a-aeb456a1f28c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:45.991736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f965e95a-ad76-4e04-af04-bd6d1c867475 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.001601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d2924-fb5b-4a37-98b5-de8a6e247fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:46.004135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d2924-fb5b-4a37-98b5-de8a6e247fa6 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.005199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d2924-fb5b-4a37-98b5-de8a6e247fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.005808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d2924-fb5b-4a37-98b5-de8a6e247fa6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.007151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d2924-fb5b-4a37-98b5-de8a6e247fa6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:46.020983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d2924-fb5b-4a37-98b5-de8a6e247fa6 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.022650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 856/1698] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:45 2022] PUT /identity/v3/projects/787051300d714b628ecbe666880d0854/users/cdf9054534b6474f88f216794ff39593/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:46.025164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeb3c26-cb33-44b7-8366-16d1b415e98e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.025780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeb3c26-cb33-44b7-8366-16d1b415e98e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.026270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeb3c26-cb33-44b7-8366-16d1b415e98e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.071006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-eeeb3c26-cb33-44b7-8366-16d1b415e98e None None] MFA Rules not processed for user `cdf9054534b6474f88f216794ff39593`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:46.072673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-eeeb3c26-cb33-44b7-8366-16d1b415e98e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:46.093108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 843/1699] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:45 2022] POST /identity/v3/projects => generated 381 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:46.098010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d2924-fb5b-4a37-98b5-de8a6e247fa6 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:46.229723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 857/1700] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:46 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:46.233478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeeb3c26-cb33-44b7-8366-16d1b415e98e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:46.277363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d2924-fb5b-4a37-98b5-de8a6e247fa6 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:46.301067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed9df0-a066-4894-9cd2-4130fbdbc369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:46.301067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed9df0-a066-4894-9cd2-4130fbdbc369 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.301067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed9df0-a066-4894-9cd2-4130fbdbc369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.301067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed9df0-a066-4894-9cd2-4130fbdbc369 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.301439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ed9df0-a066-4894-9cd2-4130fbdbc369 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:46.301511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:46.301979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ed9df0-a066-4894-9cd2-4130fbdbc369 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.376631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 844/1701] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:46 2022] POST /identity/v3/users => generated 297 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:09:46.377498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed9df0-a066-4894-9cd2-4130fbdbc369 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:46.476738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed9df0-a066-4894-9cd2-4130fbdbc369 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:46.485609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeeb3c26-cb33-44b7-8366-16d1b415e98e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:46.486769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:46.487474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.487792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.488072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.494303 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:46.501370 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912b4832-44a6-4e20-b3cd-ed938aac4f17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:46.510694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912b4832-44a6-4e20-b3cd-ed938aac4f17 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.515363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912b4832-44a6-4e20-b3cd-ed938aac4f17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.515835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912b4832-44a6-4e20-b3cd-ed938aac4f17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.519579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912b4832-44a6-4e20-b3cd-ed938aac4f17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:46.527529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912b4832-44a6-4e20-b3cd-ed938aac4f17 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.614423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:46.634658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 858/1702] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:46 2022] POST /identity/v3/projects => generated 393 bytes in 401 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:46.635902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912b4832-44a6-4e20-b3cd-ed938aac4f17 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:46.664513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:46.668113 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.718922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 845/1703] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:46 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:46.721670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:46.760198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dad8fe-0dbd-4972-b41e-0b06c1b78bd1 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:46.766523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912b4832-44a6-4e20-b3cd-ed938aac4f17 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:46.772868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:46.774204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.774690 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.775219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.776249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:46.778308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.785606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb678d6e-1fda-49e7-91be-82d41f34d25f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:46.786645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb678d6e-1fda-49e7-91be-82d41f34d25f admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.787041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb678d6e-1fda-49e7-91be-82d41f34d25f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.787507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb678d6e-1fda-49e7-91be-82d41f34d25f admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.790218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb678d6e-1fda-49e7-91be-82d41f34d25f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:46.796230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb678d6e-1fda-49e7-91be-82d41f34d25f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.800386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 859/1704] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:46 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:46.801512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:46.803440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:46.833432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269eb536-ca51-488d-b04d-05978d7b164f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:46.847904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 846/1705] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:46 2022] POST /identity/v3/users => generated 303 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:46.848387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:46.848975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb678d6e-1fda-49e7-91be-82d41f34d25f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:46.850478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.850790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.851075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] PATH_INFO: `/v3/projects/08233653cc71488393a34f602b7fd885` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.854722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=08233653cc71488393a34f602b7fd885)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:46.867231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.873494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb678d6e-1fda-49e7-91be-82d41f34d25f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:46.888046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddb16ca-844e-4e75-98ae-190f227d35c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:46.889141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb16ca-844e-4e75-98ae-190f227d35c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:46.889635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb16ca-844e-4e75-98ae-190f227d35c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:46.889986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb16ca-844e-4e75-98ae-190f227d35c1 admin admin] PATH_INFO: `/v3/projects/8dc2c9f121c348e79ad88c6f699746d4/users/db41bee6edab42368b4316254c277adb/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:46.923196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddb16ca-844e-4e75-98ae-190f227d35c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8dc2c9f121c348e79ad88c6f699746d4, user_id=db41bee6edab42368b4316254c277adb, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:46.926788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddb16ca-844e-4e75-98ae-190f227d35c1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:46.952790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 847/1706] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:46 2022] PUT /identity/v3/projects/8dc2c9f121c348e79ad88c6f699746d4/users/db41bee6edab42368b4316254c277adb/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:46.952790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddb16ca-844e-4e75-98ae-190f227d35c1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:46.977534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08233653cc71488393a34f602b7fd885', 'request_id': 'req-be8fd216-1f61-40be-9b4c-9936796560c7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:46.998289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08233653cc71488393a34f602b7fd885', 'request_id': 'req-be8fd216-1f61-40be-9b4c-9936796560c7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:47.007163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08233653cc71488393a34f602b7fd885', 'request_id': 'req-be8fd216-1f61-40be-9b4c-9936796560c7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:47.037967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] The token cache is being invalidate because project 08233653cc71488393a34f602b7fd885 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:47.039369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:47.042924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:47.047662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 860/1707] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:46 2022] DELETE /identity/v3/projects/08233653cc71488393a34f602b7fd885 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:47.050510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:47.054449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:47.117322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:47.128668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddb16ca-844e-4e75-98ae-190f227d35c1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.247241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8fd216-1f61-40be-9b4c-9936796560c7 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.257472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63b127a-81f7-48a5-ba1d-32792e293c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.258270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63b127a-81f7-48a5-ba1d-32792e293c37 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.258656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63b127a-81f7-48a5-ba1d-32792e293c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.261342 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63b127a-81f7-48a5-ba1d-32792e293c37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.262148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63b127a-81f7-48a5-ba1d-32792e293c37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:47.273076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63b127a-81f7-48a5-ba1d-32792e293c37 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:47.276706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c34e103-4fec-4efa-9ec0-c232a49cfc90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.278055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34e103-4fec-4efa-9ec0-c232a49cfc90 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.278402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34e103-4fec-4efa-9ec0-c232a49cfc90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.278689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c34e103-4fec-4efa-9ec0-c232a49cfc90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.280181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c34e103-4fec-4efa-9ec0-c232a49cfc90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:47.284685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c34e103-4fec-4efa-9ec0-c232a49cfc90 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:47.287040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 861/1708] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:47 2022] GET /identity/v3/roles => generated 1420 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:47.288525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93233b35-714f-420e-9fc6-b3d4074fed92 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.288929 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93233b35-714f-420e-9fc6-b3d4074fed92 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.289306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93233b35-714f-420e-9fc6-b3d4074fed92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.315130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-93233b35-714f-420e-9fc6-b3d4074fed92 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:47.316526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-93233b35-714f-420e-9fc6-b3d4074fed92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:47.324347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 848/1709] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:46 2022] GET /identity/v3/roles => generated 1420 bytes in 372 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:47.325283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c34e103-4fec-4efa-9ec0-c232a49cfc90 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:47.338910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 862/1710] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:47.339815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93233b35-714f-420e-9fc6-b3d4074fed92 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:47.344941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-93233b35-714f-420e-9fc6-b3d4074fed92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:47.369557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c34e103-4fec-4efa-9ec0-c232a49cfc90 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.372755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93233b35-714f-420e-9fc6-b3d4074fed92 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.384953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43987755-560d-473b-b97a-35be604c5094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.386299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a27d4c4-96d4-4eb2-bdb8-e692accbaf15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.386831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43987755-560d-473b-b97a-35be604c5094 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.387644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43987755-560d-473b-b97a-35be604c5094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.387787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a27d4c4-96d4-4eb2-bdb8-e692accbaf15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.387962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43987755-560d-473b-b97a-35be604c5094 admin admin] PATH_INFO: `/v3/projects/8dc2c9f121c348e79ad88c6f699746d4/users/db41bee6edab42368b4316254c277adb/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.388127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a27d4c4-96d4-4eb2-bdb8-e692accbaf15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.388481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a27d4c4-96d4-4eb2-bdb8-e692accbaf15 admin admin] PATH_INFO: `/v3/projects/9bf8662b5a4041b0a1c87d34ac76b43f/users/77fcdddd580e462e9c85d0f15ad2044d/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.393833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43987755-560d-473b-b97a-35be604c5094 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8dc2c9f121c348e79ad88c6f699746d4, user_id=db41bee6edab42368b4316254c277adb, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:47.400040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43987755-560d-473b-b97a-35be604c5094 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:47.429729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a27d4c4-96d4-4eb2-bdb8-e692accbaf15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9bf8662b5a4041b0a1c87d34ac76b43f, user_id=77fcdddd580e462e9c85d0f15ad2044d, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:47.434348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 849/1711] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:47 2022] PUT /identity/v3/projects/8dc2c9f121c348e79ad88c6f699746d4/users/db41bee6edab42368b4316254c277adb/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:47.437294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a27d4c4-96d4-4eb2-bdb8-e692accbaf15 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:47.437543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8e1ff9-9395-407e-9789-c4b9424c0e2e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.437887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8e1ff9-9395-407e-9789-c4b9424c0e2e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.439299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8e1ff9-9395-407e-9789-c4b9424c0e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.472887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 863/1712] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:47 2022] PUT /identity/v3/projects/9bf8662b5a4041b0a1c87d34ac76b43f/users/77fcdddd580e462e9c85d0f15ad2044d/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:47.474620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f956bdd-edcd-4671-bfeb-592951833c7e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.474908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f956bdd-edcd-4671-bfeb-592951833c7e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.475178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f956bdd-edcd-4671-bfeb-592951833c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.497685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-fd8e1ff9-9395-407e-9789-c4b9424c0e2e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:47.498755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fd8e1ff9-9395-407e-9789-c4b9424c0e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:47.554454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3f956bdd-edcd-4671-bfeb-592951833c7e None None] MFA Rules not processed for user `db41bee6edab42368b4316254c277adb`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:47.555711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3f956bdd-edcd-4671-bfeb-592951833c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:47.588759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:47.646983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 850/1713] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:47.651142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8e1ff9-9395-407e-9789-c4b9424c0e2e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:47.663005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8e1ff9-9395-407e-9789-c4b9424c0e2e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.667939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aa6f8-bb43-42fe-b1a7-a2375beb5755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.668555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aa6f8-bb43-42fe-b1a7-a2375beb5755 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.668839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aa6f8-bb43-42fe-b1a7-a2375beb5755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.669115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aa6f8-bb43-42fe-b1a7-a2375beb5755 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.669765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30aa6f8-bb43-42fe-b1a7-a2375beb5755 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:47.671446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30aa6f8-bb43-42fe-b1a7-a2375beb5755 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:47.691234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 851/1714] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:47 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:47.693324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aa6f8-bb43-42fe-b1a7-a2375beb5755 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:47.694721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 864/1715] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:47 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:47.695657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f956bdd-edcd-4671-bfeb-592951833c7e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:47.705493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aa6f8-bb43-42fe-b1a7-a2375beb5755 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.710253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.710874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.711664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.711829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.712496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:47.712803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:47.714881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:47.727167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f956bdd-edcd-4671-bfeb-592951833c7e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.737943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 852/1716] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:47.739381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:47.741015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4ef62-63ab-4d72-8ee6-83fb4c8c6f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.741154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:47.741703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4ef62-63ab-4d72-8ee6-83fb4c8c6f2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.741961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4ef62-63ab-4d72-8ee6-83fb4c8c6f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.742252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4ef62-63ab-4d72-8ee6-83fb4c8c6f2d admin admin] PATH_INFO: `/v3/projects/9bf8662b5a4041b0a1c87d34ac76b43f/users/77fcdddd580e462e9c85d0f15ad2044d/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.753471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe4ef62-63ab-4d72-8ee6-83fb4c8c6f2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9bf8662b5a4041b0a1c87d34ac76b43f, user_id=77fcdddd580e462e9c85d0f15ad2044d, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:47.768223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe4ef62-63ab-4d72-8ee6-83fb4c8c6f2d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:47.783759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 865/1717] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:47 2022] PUT /identity/v3/projects/9bf8662b5a4041b0a1c87d34ac76b43f/users/77fcdddd580e462e9c85d0f15ad2044d/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:47.784684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4ef62-63ab-4d72-8ee6-83fb4c8c6f2d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:47.836598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b3ff4c-658a-4385-ba82-6aa298a704bd admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.902351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:47.914286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4ef62-63ab-4d72-8ee6-83fb4c8c6f2d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:47.919499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d7f409-095e-4119-b80f-e2bb4c629266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.920144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7f409-095e-4119-b80f-e2bb4c629266 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.920422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7f409-095e-4119-b80f-e2bb4c629266 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.920689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7f409-095e-4119-b80f-e2bb4c629266 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.921323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d7f409-095e-4119-b80f-e2bb4c629266 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:47.927300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d7f409-095e-4119-b80f-e2bb4c629266 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:47.947907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797b6889-8114-4f8f-9807-9346832ef24e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:47.951424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797b6889-8114-4f8f-9807-9346832ef24e service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.951737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797b6889-8114-4f8f-9807-9346832ef24e service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.952016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797b6889-8114-4f8f-9807-9346832ef24e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:47.992112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 866/1718] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:47 2022] POST /identity/v3/projects => generated 361 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:47.993414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e49cc42-c5be-45cd-bf7d-69cd68a17e49 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:47.993702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e49cc42-c5be-45cd-bf7d-69cd68a17e49 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:47.993976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e49cc42-c5be-45cd-bf7d-69cd68a17e49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:48.062000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2e49cc42-c5be-45cd-bf7d-69cd68a17e49 None None] MFA Rules not processed for user `77fcdddd580e462e9c85d0f15ad2044d`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:48.064270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2e49cc42-c5be-45cd-bf7d-69cd68a17e49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:48.102281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797b6889-8114-4f8f-9807-9346832ef24e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:48.103834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797b6889-8114-4f8f-9807-9346832ef24e service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:48.128810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 853/1719] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:47 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:48.129767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797b6889-8114-4f8f-9807-9346832ef24e service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:48.152860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797b6889-8114-4f8f-9807-9346832ef24e service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:48.183718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:48.210107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 867/1720] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:47 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:48.223111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e49cc42-c5be-45cd-bf7d-69cd68a17e49 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:48.224414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2e49cc42-c5be-45cd-bf7d-69cd68a17e49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:48.294758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b499f971-97bf-40f9-af82-4b291ed7750f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:48.295391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b499f971-97bf-40f9-af82-4b291ed7750f admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:48.295669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b499f971-97bf-40f9-af82-4b291ed7750f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:48.302964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b499f971-97bf-40f9-af82-4b291ed7750f admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:48.302964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b499f971-97bf-40f9-af82-4b291ed7750f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:48.305691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b499f971-97bf-40f9-af82-4b291ed7750f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:48.362121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 854/1721] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:48 2022] POST /identity/v3/users => generated 281 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:48.366945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b499f971-97bf-40f9-af82-4b291ed7750f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:48.387004 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b499f971-97bf-40f9-af82-4b291ed7750f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:48.393731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e49cc42-c5be-45cd-bf7d-69cd68a17e49 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:48.399119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c93ceee-c907-460e-b8f0-67e343a6d0a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:48.402476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c93ceee-c907-460e-b8f0-67e343a6d0a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:48.403468 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c93ceee-c907-460e-b8f0-67e343a6d0a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:48.403468 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c93ceee-c907-460e-b8f0-67e343a6d0a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:48.404459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1b8caf-63af-4525-80a7-f874181cb667 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:48.405194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1b8caf-63af-4525-80a7-f874181cb667 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:48.405577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1b8caf-63af-4525-80a7-f874181cb667 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:48.405953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1b8caf-63af-4525-80a7-f874181cb667 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:48.406823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c93ceee-c907-460e-b8f0-67e343a6d0a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:48.408835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c93ceee-c907-460e-b8f0-67e343a6d0a7 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:48.435282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1b8caf-63af-4525-80a7-f874181cb667 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:48.437349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 855/1722] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:48.438962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1b8caf-63af-4525-80a7-f874181cb667 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:48.444045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c93ceee-c907-460e-b8f0-67e343a6d0a7 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:48.461607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c93ceee-c907-460e-b8f0-67e343a6d0a7 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:48.477063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 868/1723] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:48 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:09:48.480328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a9f161-b782-4d75-bab9-7594534644bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:48.480954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a9f161-b782-4d75-bab9-7594534644bb admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:48.481234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a9f161-b782-4d75-bab9-7594534644bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:48.481506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a9f161-b782-4d75-bab9-7594534644bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:48.483940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a9f161-b782-4d75-bab9-7594534644bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:48.487507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a9f161-b782-4d75-bab9-7594534644bb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:48.514997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 856/1724] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:48.520211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1b8caf-63af-4525-80a7-f874181cb667 service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:48.537059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1b8caf-63af-4525-80a7-f874181cb667 service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:48.549936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4004c-808c-48bd-b59f-572d0bee6e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:48.550633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4004c-808c-48bd-b59f-572d0bee6e39 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:48.550918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4004c-808c-48bd-b59f-572d0bee6e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:48.551336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4004c-808c-48bd-b59f-572d0bee6e39 admin admin] PATH_INFO: `/v3/projects/2372f682fec44246b5845703eeab5df4/users/238ebdc5126348e19b4cea1677dba8f3/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:48.589575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e4004c-808c-48bd-b59f-572d0bee6e39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2372f682fec44246b5845703eeab5df4, user_id=238ebdc5126348e19b4cea1677dba8f3, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:48.593569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e4004c-808c-48bd-b59f-572d0bee6e39 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:48.630624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 869/1725] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:48 2022] PUT /identity/v3/projects/2372f682fec44246b5845703eeab5df4/users/238ebdc5126348e19b4cea1677dba8f3/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:48.643059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3685429-ea90-44c7-9756-cebba8bea217 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:48.643345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3685429-ea90-44c7-9756-cebba8bea217 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:48.643625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3685429-ea90-44c7-9756-cebba8bea217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:48.691757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c3685429-ea90-44c7-9756-cebba8bea217 None None] MFA Rules not processed for user `238ebdc5126348e19b4cea1677dba8f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:48.693705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c3685429-ea90-44c7-9756-cebba8bea217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:48.866645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 857/1726] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:48.868298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4004c-808c-48bd-b59f-572d0bee6e39 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:48.968805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3685429-ea90-44c7-9756-cebba8bea217 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:48.976439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:49.012004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4004c-808c-48bd-b59f-572d0bee6e39 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:49.017680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6791f57a-5f89-46c2-9f06-01c5accf1218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:49.018312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6791f57a-5f89-46c2-9f06-01c5accf1218 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:49.018597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6791f57a-5f89-46c2-9f06-01c5accf1218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:49.018870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6791f57a-5f89-46c2-9f06-01c5accf1218 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:49.023433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6791f57a-5f89-46c2-9f06-01c5accf1218 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:49.025503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6791f57a-5f89-46c2-9f06-01c5accf1218 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:49.085198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 870/1727] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:48 2022] POST /identity/v3/projects => generated 365 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:49.088893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6791f57a-5f89-46c2-9f06-01c5accf1218 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:49.089219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3685429-ea90-44c7-9756-cebba8bea217 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:49.160517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:49.202691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:49.204911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:49.204911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:49.204911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:49.211333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:49.257373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6791f57a-5f89-46c2-9f06-01c5accf1218 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:49.263844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceba01e-6f8b-495c-b831-85a4ea87d379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:49.264719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceba01e-6f8b-495c-b831-85a4ea87d379 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:49.265134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceba01e-6f8b-495c-b831-85a4ea87d379 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:49.265560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceba01e-6f8b-495c-b831-85a4ea87d379 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:49.267557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceba01e-6f8b-495c-b831-85a4ea87d379 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:49.269854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceba01e-6f8b-495c-b831-85a4ea87d379 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:49.315238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 871/1728] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:49 2022] POST /identity/v3/users => generated 289 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:49.319316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceba01e-6f8b-495c-b831-85a4ea87d379 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:49.349119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:49.351150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:49.358458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceba01e-6f8b-495c-b831-85a4ea87d379 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:49.373281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a5b888-4a90-4642-8bf3-95ff09dd1693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:49.374029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a5b888-4a90-4642-8bf3-95ff09dd1693 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:49.374185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a5b888-4a90-4642-8bf3-95ff09dd1693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:49.374460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a5b888-4a90-4642-8bf3-95ff09dd1693 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:49.375654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a5b888-4a90-4642-8bf3-95ff09dd1693 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:49.376061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 858/1729] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:48 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:49.381049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a5b888-4a90-4642-8bf3-95ff09dd1693 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:49.407093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 872/1730] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:49 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:49.410962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:49.447233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a522c5b5-8cee-4357-802d-26f2d783e40b service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:49.466527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff48e7e6-cd63-435a-9c31-859a6f70b43e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:49.471157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff48e7e6-cd63-435a-9c31-859a6f70b43e admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:49.471805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff48e7e6-cd63-435a-9c31-859a6f70b43e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:49.472127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff48e7e6-cd63-435a-9c31-859a6f70b43e admin admin] PATH_INFO: `/v3/projects/38b44c1bdab64428a8ccd8c43d9492f6/users/0c6143cf4b2d469ba3c29369dc06abac/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:49.519408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff48e7e6-cd63-435a-9c31-859a6f70b43e admin admin] RBAC: Authorizing `identity:create_grant(project_id=38b44c1bdab64428a8ccd8c43d9492f6, user_id=0c6143cf4b2d469ba3c29369dc06abac, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:49.522237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff48e7e6-cd63-435a-9c31-859a6f70b43e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:49.554968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 859/1731] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:49 2022] PUT /identity/v3/projects/38b44c1bdab64428a8ccd8c43d9492f6/users/0c6143cf4b2d469ba3c29369dc06abac/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:49.556332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a5b888-4a90-4642-8bf3-95ff09dd1693 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:49.579546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a5b888-4a90-4642-8bf3-95ff09dd1693 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:49.785195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba7294c-78ff-4b74-babc-7f9711cbf359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:49.786882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba7294c-78ff-4b74-babc-7f9711cbf359 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:49.788173 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba7294c-78ff-4b74-babc-7f9711cbf359 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:49.788968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba7294c-78ff-4b74-babc-7f9711cbf359 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:49.790470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba7294c-78ff-4b74-babc-7f9711cbf359 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:49.807740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba7294c-78ff-4b74-babc-7f9711cbf359 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:49.849644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 873/1732] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:49 2022] GET /identity/v3/roles => generated 1420 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:49.850863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff48e7e6-cd63-435a-9c31-859a6f70b43e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:49.869766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff48e7e6-cd63-435a-9c31-859a6f70b43e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:49.891968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4530ed43-9419-4144-918b-c0dfee64630e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:49.892862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4530ed43-9419-4144-918b-c0dfee64630e admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:49.893360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4530ed43-9419-4144-918b-c0dfee64630e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:49.900213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4530ed43-9419-4144-918b-c0dfee64630e admin admin] PATH_INFO: `/v3/projects/38b44c1bdab64428a8ccd8c43d9492f6/users/0c6143cf4b2d469ba3c29369dc06abac/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:49.907211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4530ed43-9419-4144-918b-c0dfee64630e admin admin] RBAC: Authorizing `identity:create_grant(project_id=38b44c1bdab64428a8ccd8c43d9492f6, user_id=0c6143cf4b2d469ba3c29369dc06abac, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:49.918813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4530ed43-9419-4144-918b-c0dfee64630e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:49.966034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba7294c-78ff-4b74-babc-7f9711cbf359 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:49.977098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 860/1733] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:49 2022] PUT /identity/v3/projects/38b44c1bdab64428a8ccd8c43d9492f6/users/0c6143cf4b2d469ba3c29369dc06abac/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:49.991032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5105b0-e165-4590-bf9a-dd2d7b2b90f2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:49.991644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5105b0-e165-4590-bf9a-dd2d7b2b90f2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:49.993525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5105b0-e165-4590-bf9a-dd2d7b2b90f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:50.175663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6b5105b0-e165-4590-bf9a-dd2d7b2b90f2 None None] MFA Rules not processed for user `0c6143cf4b2d469ba3c29369dc06abac`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:50.177097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6b5105b0-e165-4590-bf9a-dd2d7b2b90f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:50.194936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:50.273425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba7294c-78ff-4b74-babc-7f9711cbf359 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:50.434137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 861/1734] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:49 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 448 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:50.455547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5105b0-e165-4590-bf9a-dd2d7b2b90f2 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:50.539988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db41bee6edab42368b4316254c277adb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8dc2c9f121c348e79ad88c6f699746d4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:50.540625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:50.540918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:50.541199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:50.541872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:50.542184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:50.543851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:50.549002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:50.572643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 874/1735] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 607 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:50.578837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:50.602159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8ad54e-3a42-4022-a932-fc8e3082422b tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:50.608389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5105b0-e165-4590-bf9a-dd2d7b2b90f2 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:50.620371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316e152d-e71f-46eb-ab65-ea61413c2cd6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db41bee6edab42368b4316254c277adb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8dc2c9f121c348e79ad88c6f699746d4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:50.622617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e152d-e71f-46eb-ab65-ea61413c2cd6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:50.623170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e152d-e71f-46eb-ab65-ea61413c2cd6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:50.624192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e152d-e71f-46eb-ab65-ea61413c2cd6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:50.625809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd218361-5d09-4850-b939-a736c7997b4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:50.627434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316e152d-e71f-46eb-ab65-ea61413c2cd6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:50.628365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd218361-5d09-4850-b939-a736c7997b4a service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:50.630354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316e152d-e71f-46eb-ab65-ea61413c2cd6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:50.631285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd218361-5d09-4850-b939-a736c7997b4a service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:50.631285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd218361-5d09-4850-b939-a736c7997b4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:50.659164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd218361-5d09-4850-b939-a736c7997b4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:50.661460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd218361-5d09-4850-b939-a736c7997b4a service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:50.686258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 862/1736] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:50 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:50.698444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 875/1737] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:50 2022] POST /identity/v3/projects => generated 354 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:50.700154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd218361-5d09-4850-b939-a736c7997b4a service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:50.720653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd218361-5d09-4850-b939-a736c7997b4a service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:50.843395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316e152d-e71f-46eb-ab65-ea61413c2cd6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:50.878106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db41bee6edab42368b4316254c277adb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8dc2c9f121c348e79ad88c6f699746d4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:50.879330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:50.882970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:50.883309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:50.884167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:50.884719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:50.886464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:50.923901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 863/1738] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 14:09:51.035069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316e152d-e71f-46eb-ab65-ea61413c2cd6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:51.072791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:51.088761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857acae1-2975-4cab-9ccc-13d102ecb619 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:51.089565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857acae1-2975-4cab-9ccc-13d102ecb619 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:51.089991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857acae1-2975-4cab-9ccc-13d102ecb619 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:51.092741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857acae1-2975-4cab-9ccc-13d102ecb619 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:51.102563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad333f14-16a7-4990-906c-07ccaeffadb6 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:51.113907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db41bee6edab42368b4316254c277adb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8dc2c9f121c348e79ad88c6f699746d4', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:51.114876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:51.115441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:51.115981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] PATH_INFO: `/v3/projects/5a32fdb3b9a445f2b553c6329dd7ae58` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:51.117515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Authorizing `identity:delete_project(project_id=5a32fdb3b9a445f2b553c6329dd7ae58)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:51.120443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:51.270436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a32fdb3b9a445f2b553c6329dd7ae58', 'request_id': 'req-fe66ce5f-8705-4b25-ba10-2d037f10fab3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:51.284277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857acae1-2975-4cab-9ccc-13d102ecb619 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:51.287097 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857acae1-2975-4cab-9ccc-13d102ecb619 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:51.298908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a32fdb3b9a445f2b553c6329dd7ae58', 'request_id': 'req-fe66ce5f-8705-4b25-ba10-2d037f10fab3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:51.315210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 876/1739] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:50 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:51.322975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a32fdb3b9a445f2b553c6329dd7ae58', 'request_id': 'req-fe66ce5f-8705-4b25-ba10-2d037f10fab3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:51.358208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] The token cache is being invalidate because project 5a32fdb3b9a445f2b553c6329dd7ae58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:51.358741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:51.360990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:51.364217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 864/1740] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:51 2022] DELETE /identity/v3/projects/5a32fdb3b9a445f2b553c6329dd7ae58 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:52.197080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fa330-3191-4bc0-b523-b5b7cb5b3daa None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:52.197371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fa330-3191-4bc0-b523-b5b7cb5b3daa None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:52.197645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9fa330-3191-4bc0-b523-b5b7cb5b3daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:52.282987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ba9fa330-3191-4bc0-b523-b5b7cb5b3daa None None] MFA Rules not processed for user `f521b038310940d1b5ee7fc5afc1cbac`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:52.283531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ba9fa330-3191-4bc0-b523-b5b7cb5b3daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:52.321467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:52.436867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:52.438887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:52.529354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 877/1741] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:52.540748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387e4308-ad2e-4fb2-b3db-50d088357f4e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:52.541177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387e4308-ad2e-4fb2-b3db-50d088357f4e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:52.542295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387e4308-ad2e-4fb2-b3db-50d088357f4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:52.596411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:52.597804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-387e4308-ad2e-4fb2-b3db-50d088357f4e None None] MFA Rules not processed for user `f521b038310940d1b5ee7fc5afc1cbac`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:52.603998 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-387e4308-ad2e-4fb2-b3db-50d088357f4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:52.621875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66ce5f-8705-4b25-ba10-2d037f10fab3 tempest-VolumeQuotaClassesTest-1234956577 tempest-VolumeQuotaClassesTest-1234956577-project-admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:52.635191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:52.635894 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:52.636168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:52.636503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] PATH_INFO: `/v3/users/c400909b5c434bfbb3686107b570dc26` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:52.637710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] RBAC: Authorizing `identity:delete_user(user_id=c400909b5c434bfbb3686107b570dc26)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:52.640581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 878/1742] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:52.644270 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:52.650554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387e4308-ad2e-4fb2-b3db-50d088357f4e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:52.652229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-387e4308-ad2e-4fb2-b3db-50d088357f4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:52.768036 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c400909b5c434bfbb3686107b570dc26'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:52.779525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c400909b5c434bfbb3686107b570dc26'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:52.780263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c400909b5c434bfbb3686107b570dc26'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:52.780904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c400909b5c434bfbb3686107b570dc26'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:52.820757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3688c4c4-a222-406d-bf0c-4c0ce66e6dce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c400909b5c434bfbb3686107b570dc26'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:52.919635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 865/1743] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:09:52 2022] DELETE /identity/v3/users/c400909b5c434bfbb3686107b570dc26 => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:52.921210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe156ef3-216c-46bd-8aa2-7931c63f275e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:52.921498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe156ef3-216c-46bd-8aa2-7931c63f275e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:52.921779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe156ef3-216c-46bd-8aa2-7931c63f275e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:52.957562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387e4308-ad2e-4fb2-b3db-50d088357f4e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:52.991551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-fe156ef3-216c-46bd-8aa2-7931c63f275e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:52.993002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fe156ef3-216c-46bd-8aa2-7931c63f275e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:53.196035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 866/1744] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:53.207406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe156ef3-216c-46bd-8aa2-7931c63f275e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:53.209116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fe156ef3-216c-46bd-8aa2-7931c63f275e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:53.220355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:53.221834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:53.224131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:53.224131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:53.231972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:53.365866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:53.372627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe156ef3-216c-46bd-8aa2-7931c63f275e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:53.385674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568c742f-25e3-444e-99ee-cbec3a01f07e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:53.386524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568c742f-25e3-444e-99ee-cbec3a01f07e service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:53.387645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568c742f-25e3-444e-99ee-cbec3a01f07e service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:53.388432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568c742f-25e3-444e-99ee-cbec3a01f07e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:53.419087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568c742f-25e3-444e-99ee-cbec3a01f07e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:53.422180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568c742f-25e3-444e-99ee-cbec3a01f07e service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:53.428700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:53.430778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:53.452560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 867/1745] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:53.454099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 879/1746] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:52 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 804 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:53.596266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa57b0c-5816-4c1d-8a41-77235814f05c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:53.596592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa57b0c-5816-4c1d-8a41-77235814f05c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:53.596826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa57b0c-5816-4c1d-8a41-77235814f05c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:53.621101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-0aa57b0c-5816-4c1d-8a41-77235814f05c None None] MFA Rules not processed for user `2049d66708ce47a794c14ad4c2040937`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:53.622349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0aa57b0c-5816-4c1d-8a41-77235814f05c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:53.639930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:53.784907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 868/1747] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:53 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:54.241081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:54.243269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:54.269873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b959ed-ddd0-4fb3-bd76-2525ded0b01b service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:54.283849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:54.284578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:54.284867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:54.285144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] PATH_INFO: `/v3/projects/1c79952eb7644ad2842f103ee5ff9e20` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:54.286803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c79952eb7644ad2842f103ee5ff9e20)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:54.290652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:54.422456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c79952eb7644ad2842f103ee5ff9e20', 'request_id': 'req-16cee2f6-6ea9-4133-826e-e541055dd9c1'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.449456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c79952eb7644ad2842f103ee5ff9e20', 'request_id': 'req-16cee2f6-6ea9-4133-826e-e541055dd9c1'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.477663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c79952eb7644ad2842f103ee5ff9e20', 'request_id': 'req-16cee2f6-6ea9-4133-826e-e541055dd9c1'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.497943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] The token cache is being invalidate because project 1c79952eb7644ad2842f103ee5ff9e20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:54.498422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.504802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16cee2f6-6ea9-4133-826e-e541055dd9c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.515275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 880/1748] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:54 2022] DELETE /identity/v3/projects/1c79952eb7644ad2842f103ee5ff9e20 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:54.520655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa57b0c-5816-4c1d-8a41-77235814f05c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:54.552308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa57b0c-5816-4c1d-8a41-77235814f05c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:54.711488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:54.711488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:54.711972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:54.712258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] PATH_INFO: `/v3/users/db41bee6edab42368b4316254c277adb` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:54.714020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] RBAC: Authorizing `identity:delete_user(user_id=db41bee6edab42368b4316254c277adb)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:54.716060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:54.725046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc2b2f-2a4e-4447-b268-82803ae6aed7 None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:54.725620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc2b2f-2a4e-4447-b268-82803ae6aed7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:54.725898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc2b2f-2a4e-4447-b268-82803ae6aed7 None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:54.727412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 881/1749] 10.222.0.27 () {54 vars in 912 bytes} [Thu Sep 15 14:09:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 15 14:09:54.737667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92a1f8d-089d-49bb-aa9d-1a5136ae7cc3 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:54.737976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92a1f8d-089d-49bb-aa9d-1a5136ae7cc3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:54.738244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92a1f8d-089d-49bb-aa9d-1a5136ae7cc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:54.807347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d92a1f8d-089d-49bb-aa9d-1a5136ae7cc3 None None] MFA Rules not processed for user `922012eacf244f9480c180c2eb9eaf28`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:54.809047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d92a1f8d-089d-49bb-aa9d-1a5136ae7cc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:54.824256 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db41bee6edab42368b4316254c277adb'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.837057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db41bee6edab42368b4316254c277adb'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.837835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db41bee6edab42368b4316254c277adb'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.838638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db41bee6edab42368b4316254c277adb'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.857273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-be46aeaf-57fb-499a-a44d-a03d67f47822 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db41bee6edab42368b4316254c277adb'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:54.898285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 869/1750] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:09:54 2022] DELETE /identity/v3/users/db41bee6edab42368b4316254c277adb => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:54.908668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43816c81-49ae-40a0-bf36-97412b5bacf5 None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:54.908668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43816c81-49ae-40a0-bf36-97412b5bacf5 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:54.908668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43816c81-49ae-40a0-bf36-97412b5bacf5 None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:54.908668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 870/1751] 10.222.0.27 () {54 vars in 912 bytes} [Thu Sep 15 14:09:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:09:54.924840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80c396-3534-4afb-9dcf-62d4e856f0ce None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:54.925783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80c396-3534-4afb-9dcf-62d4e856f0ce None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:54.925783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80c396-3534-4afb-9dcf-62d4e856f0ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:54.963260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 882/1752] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:09:54 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:54.963892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6b80c396-3534-4afb-9dcf-62d4e856f0ce None None] MFA Rules not processed for user `922012eacf244f9480c180c2eb9eaf28`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:54.965176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6b80c396-3534-4afb-9dcf-62d4e856f0ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:55.008558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0220f802-2048-4871-8eb9-64e9cc640bcb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:55.008851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0220f802-2048-4871-8eb9-64e9cc640bcb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:55.009124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0220f802-2048-4871-8eb9-64e9cc640bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:55.051602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:55.095100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 871/1753] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:09:54 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:55.096252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b80c396-3534-4afb-9dcf-62d4e856f0ce None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:55.097561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6b80c396-3534-4afb-9dcf-62d4e856f0ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:55.110056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-0220f802-2048-4871-8eb9-64e9cc640bcb None None] MFA Rules not processed for user `530e6587b39b4f4c98d7414b81453628`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:55.113534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b80c396-3534-4afb-9dcf-62d4e856f0ce None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:55.114628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0220f802-2048-4871-8eb9-64e9cc640bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:55.118018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:55.124164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:55.124390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:55.127987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:55.166302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:55.171452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:55.206648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 872/1754] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:55.207702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:55.213992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:55.238564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:55.256754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fbcfe6-bfed-436b-a5be-59d054c9df8b service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:55.268108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:55.271180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:55.272757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:55.273053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:55.337662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 883/1755] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:55 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 327 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:55.339269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a241b58a-0c12-4b21-825b-47b75139f75b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:55.339748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a241b58a-0c12-4b21-825b-47b75139f75b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:55.339861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a241b58a-0c12-4b21-825b-47b75139f75b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:55.341029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:55.342093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:55.371717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 873/1756] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:55.395765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a241b58a-0c12-4b21-825b-47b75139f75b None None] MFA Rules not processed for user `530e6587b39b4f4c98d7414b81453628`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:55.396880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a241b58a-0c12-4b21-825b-47b75139f75b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:55.423625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 884/1757] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:55 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:55.963112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:55.964475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.088375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a241b58a-0c12-4b21-825b-47b75139f75b None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:56.090179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a241b58a-0c12-4b21-825b-47b75139f75b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.131981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8872730-63a8-4f1e-8319-0008d663012d service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:56.144942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:56.146165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.146672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.147136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] PATH_INFO: `/v3/projects/8dc2c9f121c348e79ad88c6f699746d4` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.149628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] RBAC: Authorizing `identity:delete_project(project_id=8dc2c9f121c348e79ad88c6f699746d4)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:56.163555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:56.288330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8dc2c9f121c348e79ad88c6f699746d4', 'request_id': 'req-2ee1bd66-46f9-439e-a889-ef0f28ed663c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:56.300870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8dc2c9f121c348e79ad88c6f699746d4', 'request_id': 'req-2ee1bd66-46f9-439e-a889-ef0f28ed663c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:56.320187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8dc2c9f121c348e79ad88c6f699746d4', 'request_id': 'req-2ee1bd66-46f9-439e-a889-ef0f28ed663c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:56.320900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a241b58a-0c12-4b21-825b-47b75139f75b None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:56.362198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] The token cache is being invalidate because project 8dc2c9f121c348e79ad88c6f699746d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:09:56.362433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:56.363086 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:09:56.368940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 874/1758] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:09:55 2022] DELETE /identity/v3/projects/8dc2c9f121c348e79ad88c6f699746d4 => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:56.369828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:56.374214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.495914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545ad5b4-74ca-43e4-a2c5-36ab7016fbc1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:56.501663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545ad5b4-74ca-43e4-a2c5-36ab7016fbc1 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.503274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545ad5b4-74ca-43e4-a2c5-36ab7016fbc1 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.504017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545ad5b4-74ca-43e4-a2c5-36ab7016fbc1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.514343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:56.520063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-545ad5b4-74ca-43e4-a2c5-36ab7016fbc1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.554904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee1bd66-46f9-439e-a889-ef0f28ed663c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:56.569588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92e1efd-2826-488d-83ae-e2f66a2046d7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:56.570440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92e1efd-2826-488d-83ae-e2f66a2046d7 service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.570824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92e1efd-2826-488d-83ae-e2f66a2046d7 service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.571203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92e1efd-2826-488d-83ae-e2f66a2046d7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.574732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c92e1efd-2826-488d-83ae-e2f66a2046d7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.591828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92e1efd-2826-488d-83ae-e2f66a2046d7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:56.594328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92e1efd-2826-488d-83ae-e2f66a2046d7 service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:56.607171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 875/1759] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:56.608723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ffda97-d01e-4773-bd81-d26917183958 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.609109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ffda97-d01e-4773-bd81-d26917183958 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.609666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ffda97-d01e-4773-bd81-d26917183958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.636613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545ad5b4-74ca-43e4-a2c5-36ab7016fbc1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:56.639591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545ad5b4-74ca-43e4-a2c5-36ab7016fbc1 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:56.647295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f7ffda97-d01e-4773-bd81-d26917183958 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:56.649322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f7ffda97-d01e-4773-bd81-d26917183958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.678452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 885/1760] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:56 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 590 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:09:56.684388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48cc68b-6a74-4548-8df5-6d899fb819bd None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.684715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48cc68b-6a74-4548-8df5-6d899fb819bd None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.685001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48cc68b-6a74-4548-8df5-6d899fb819bd None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.694668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 886/1761] 10.222.0.27 () {54 vars in 961 bytes} [Thu Sep 15 14:09:56 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:09:56.696085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a104eb-f5e9-4053-8cbf-d3752db54e8a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.696372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a104eb-f5e9-4053-8cbf-d3752db54e8a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.696645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a104eb-f5e9-4053-8cbf-d3752db54e8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.739632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 876/1762] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:56.743979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ec87b4-d064-4137-9aac-dc3e6b806b33 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.744274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ec87b4-d064-4137-9aac-dc3e6b806b33 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.744552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ec87b4-d064-4137-9aac-dc3e6b806b33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.768619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-60ec87b4-d064-4137-9aac-dc3e6b806b33 None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:56.769849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-60ec87b4-d064-4137-9aac-dc3e6b806b33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.775653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-23a104eb-f5e9-4053-8cbf-d3752db54e8a None None] MFA Rules not processed for user `00374d3d3bf649749781792273e35e3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:56.780214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-23a104eb-f5e9-4053-8cbf-d3752db54e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.786170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 877/1763] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Sep 15 14:09:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:56.787568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b298a5-c501-49a0-a070-4093bbf38dd6 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.787891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b298a5-c501-49a0-a070-4093bbf38dd6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.788164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b298a5-c501-49a0-a070-4093bbf38dd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.812016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:56.843765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-55b298a5-c501-49a0-a070-4093bbf38dd6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:56.843765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-55b298a5-c501-49a0-a070-4093bbf38dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:56.850986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 878/1764] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:56.852558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b298a5-c501-49a0-a070-4093bbf38dd6 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:56.867524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b298a5-c501-49a0-a070-4093bbf38dd6 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:56.873259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc13705-1e82-460e-87f2-e31d823225bf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:56.874456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc13705-1e82-460e-87f2-e31d823225bf service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.874825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc13705-1e82-460e-87f2-e31d823225bf service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.875781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc13705-1e82-460e-87f2-e31d823225bf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.889506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc13705-1e82-460e-87f2-e31d823225bf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:56.892434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc13705-1e82-460e-87f2-e31d823225bf service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:56.927028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 879/1765] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:56.929542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc13705-1e82-460e-87f2-e31d823225bf service placement] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:56.956543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc13705-1e82-460e-87f2-e31d823225bf service placement] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:56.969996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:56.970694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:56.971541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:56.972729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:56.975014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:09:56.975429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:56.978853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:57.000797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 887/1766] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:56 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:57.017126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a104eb-f5e9-4053-8cbf-d3752db54e8a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:57.018558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-23a104eb-f5e9-4053-8cbf-d3752db54e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:57.038467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a104eb-f5e9-4053-8cbf-d3752db54e8a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:57.060277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 880/1767] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:09:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:09:57.062113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16edaa4-8e9e-4f2c-b114-49246cff890e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:57.062797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16edaa4-8e9e-4f2c-b114-49246cff890e service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:57.063075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16edaa4-8e9e-4f2c-b114-49246cff890e service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:57.063459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16edaa4-8e9e-4f2c-b114-49246cff890e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:57.065584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:57.073044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16edaa4-8e9e-4f2c-b114-49246cff890e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:57.074655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16edaa4-8e9e-4f2c-b114-49246cff890e service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:57.096483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 888/1768] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:57 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:09:57.097143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b306427-6744-4954-8060-cf6d1f9b4a60 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:57.104413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b23596-cadd-47b9-ad0c-4f9000634e4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:57.105260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b23596-cadd-47b9-ad0c-4f9000634e4e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:57.105774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b23596-cadd-47b9-ad0c-4f9000634e4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:57.106234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b23596-cadd-47b9-ad0c-4f9000634e4e admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:57.107263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b23596-cadd-47b9-ad0c-4f9000634e4e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:57.110398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b23596-cadd-47b9-ad0c-4f9000634e4e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:57.156061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 881/1769] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:57 2022] POST /identity/v3/projects => generated 383 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:09:57.160771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16edaa4-8e9e-4f2c-b114-49246cff890e service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:57.182373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16edaa4-8e9e-4f2c-b114-49246cff890e service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:57.356479 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6166d0a5-8adb-4b9b-926f-5412d34afeb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:57.357088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6166d0a5-8adb-4b9b-926f-5412d34afeb3 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:57.357429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6166d0a5-8adb-4b9b-926f-5412d34afeb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:57.357708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6166d0a5-8adb-4b9b-926f-5412d34afeb3 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:57.364969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6166d0a5-8adb-4b9b-926f-5412d34afeb3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:57.369070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6166d0a5-8adb-4b9b-926f-5412d34afeb3 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:57.413491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 889/1770] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:57 2022] POST /identity/v3/users => generated 292 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:57.417499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b23596-cadd-47b9-ad0c-4f9000634e4e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:57.431452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b23596-cadd-47b9-ad0c-4f9000634e4e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:57.443443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76c98eb-e063-4da8-aea7-7917be4b251a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:57.444110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76c98eb-e063-4da8-aea7-7917be4b251a admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:57.444382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76c98eb-e063-4da8-aea7-7917be4b251a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:57.444723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76c98eb-e063-4da8-aea7-7917be4b251a admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:57.445385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76c98eb-e063-4da8-aea7-7917be4b251a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:57.446990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76c98eb-e063-4da8-aea7-7917be4b251a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:57.473946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 882/1771] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:57 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:57.478104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6166d0a5-8adb-4b9b-926f-5412d34afeb3 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:57.491971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6166d0a5-8adb-4b9b-926f-5412d34afeb3 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:57.503604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b73dfdf-dce2-4622-b887-5e7166b95fe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:57.504940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b73dfdf-dce2-4622-b887-5e7166b95fe2 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:57.505345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b73dfdf-dce2-4622-b887-5e7166b95fe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:57.505621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b73dfdf-dce2-4622-b887-5e7166b95fe2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:57.506275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b73dfdf-dce2-4622-b887-5e7166b95fe2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:57.514623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b73dfdf-dce2-4622-b887-5e7166b95fe2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:57.550129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 890/1772] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:57 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:57.554835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76c98eb-e063-4da8-aea7-7917be4b251a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:57.575409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76c98eb-e063-4da8-aea7-7917be4b251a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:57.589701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59360172-7846-4762-9b56-42eb4f98fb04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:57.590670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59360172-7846-4762-9b56-42eb4f98fb04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:57.591093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59360172-7846-4762-9b56-42eb4f98fb04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:57.591517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59360172-7846-4762-9b56-42eb4f98fb04 admin admin] PATH_INFO: `/v3/projects/26eec6155e2d45f78968cd98e9b4226c/users/426ee1c4c0cb44cc8b33cfe29e9dfaa8/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:57.644828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59360172-7846-4762-9b56-42eb4f98fb04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26eec6155e2d45f78968cd98e9b4226c, user_id=426ee1c4c0cb44cc8b33cfe29e9dfaa8, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:57.651288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59360172-7846-4762-9b56-42eb4f98fb04 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:57.688987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 883/1773] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:57 2022] PUT /identity/v3/projects/26eec6155e2d45f78968cd98e9b4226c/users/426ee1c4c0cb44cc8b33cfe29e9dfaa8/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:09:57.699850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db5321b-9869-4835-9a9f-ef28d75653ed None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:57.700748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db5321b-9869-4835-9a9f-ef28d75653ed None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:57.702298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db5321b-9869-4835-9a9f-ef28d75653ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:57.813219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8db5321b-9869-4835-9a9f-ef28d75653ed None None] MFA Rules not processed for user `426ee1c4c0cb44cc8b33cfe29e9dfaa8`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:57.818313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8db5321b-9869-4835-9a9f-ef28d75653ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:58.059211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:58.081173 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 891/1774] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 383 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:58.085839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59360172-7846-4762-9b56-42eb4f98fb04 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:58.176343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:09:58.198492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59360172-7846-4762-9b56-42eb4f98fb04 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:58.207204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51df4d33-e352-45fc-a022-b81427ea1d27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:58.208163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51df4d33-e352-45fc-a022-b81427ea1d27 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:58.208645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51df4d33-e352-45fc-a022-b81427ea1d27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:58.209118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51df4d33-e352-45fc-a022-b81427ea1d27 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:58.210922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51df4d33-e352-45fc-a022-b81427ea1d27 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:58.212495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51df4d33-e352-45fc-a022-b81427ea1d27 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:58.246983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624e03e7-4bef-4e81-9af1-2e625725dd5d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:58.247674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624e03e7-4bef-4e81-9af1-2e625725dd5d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:58.248177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624e03e7-4bef-4e81-9af1-2e625725dd5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:58.272367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 884/1775] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:09:58 2022] POST /identity/v3/projects => generated 383 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:58.276350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51df4d33-e352-45fc-a022-b81427ea1d27 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:58.291928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-624e03e7-4bef-4e81-9af1-2e625725dd5d None None] MFA Rules not processed for user `530e6587b39b4f4c98d7414b81453628`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:58.300997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-624e03e7-4bef-4e81-9af1-2e625725dd5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:58.474191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51df4d33-e352-45fc-a022-b81427ea1d27 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:58.480209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251d8848-af3e-4b01-951e-e283fbbf26f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:58.481802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d8848-af3e-4b01-951e-e283fbbf26f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:58.482291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d8848-af3e-4b01-951e-e283fbbf26f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:58.482806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251d8848-af3e-4b01-951e-e283fbbf26f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:58.483799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251d8848-af3e-4b01-951e-e283fbbf26f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:58.486866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251d8848-af3e-4b01-951e-e283fbbf26f8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:58.496408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 892/1776] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:58 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:09:58.547557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 885/1777] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:09:58 2022] POST /identity/v3/users => generated 298 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:09:58.552318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624e03e7-4bef-4e81-9af1-2e625725dd5d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:58.588156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624e03e7-4bef-4e81-9af1-2e625725dd5d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:58.603482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b37c1c-766b-4692-b861-16d86992d226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:58.605169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b37c1c-766b-4692-b861-16d86992d226 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:58.605482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b37c1c-766b-4692-b861-16d86992d226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:58.606446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b37c1c-766b-4692-b861-16d86992d226 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:58.606446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b37c1c-766b-4692-b861-16d86992d226 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:58.608279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b37c1c-766b-4692-b861-16d86992d226 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:58.642307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 893/1778] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:58 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:58.646154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251d8848-af3e-4b01-951e-e283fbbf26f8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:58.660821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251d8848-af3e-4b01-951e-e283fbbf26f8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:58.676214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847ff1b-4cb0-4ed3-9f4c-61efc7b07681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:58.676895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847ff1b-4cb0-4ed3-9f4c-61efc7b07681 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:58.677174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847ff1b-4cb0-4ed3-9f4c-61efc7b07681 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:58.677446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847ff1b-4cb0-4ed3-9f4c-61efc7b07681 admin admin] PATH_INFO: `/v3/projects/9465e3a9227145e4823542de34777eae/users/018335d2829640e1bda1e3c751ae46f2/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:58.721100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0847ff1b-4cb0-4ed3-9f4c-61efc7b07681 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9465e3a9227145e4823542de34777eae, user_id=018335d2829640e1bda1e3c751ae46f2, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:58.724923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0847ff1b-4cb0-4ed3-9f4c-61efc7b07681 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:58.751898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 886/1779] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:58 2022] PUT /identity/v3/projects/9465e3a9227145e4823542de34777eae/users/018335d2829640e1bda1e3c751ae46f2/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:58.753282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b37c1c-766b-4692-b861-16d86992d226 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:58.770468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b37c1c-766b-4692-b861-16d86992d226 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:58.901894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e17c58-6f60-4d9e-8649-bf0f47d8eae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:58.902780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e17c58-6f60-4d9e-8649-bf0f47d8eae0 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:58.903139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e17c58-6f60-4d9e-8649-bf0f47d8eae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:58.903430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e17c58-6f60-4d9e-8649-bf0f47d8eae0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:58.906023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e17c58-6f60-4d9e-8649-bf0f47d8eae0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:58.916044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e17c58-6f60-4d9e-8649-bf0f47d8eae0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:58.952528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 894/1780] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:09:58 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:09:58.954720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847ff1b-4cb0-4ed3-9f4c-61efc7b07681 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:58.971306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847ff1b-4cb0-4ed3-9f4c-61efc7b07681 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:58.988891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:58.989744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:58.990154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:58.990597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] PATH_INFO: `/v3/projects/9465e3a9227145e4823542de34777eae/users/018335d2829640e1bda1e3c751ae46f2/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:58.995547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9465e3a9227145e4823542de34777eae, user_id=018335d2829640e1bda1e3c751ae46f2, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:58.998365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:59.039771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 887/1781] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:09:58 2022] PUT /identity/v3/projects/9465e3a9227145e4823542de34777eae/users/018335d2829640e1bda1e3c751ae46f2/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:09:59.042517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ec0c15-677f-4a13-9a5a-664cac7146bd None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:59.042822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ec0c15-677f-4a13-9a5a-664cac7146bd None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:59.043093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ec0c15-677f-4a13-9a5a-664cac7146bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:59.107476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-91ec0c15-677f-4a13-9a5a-664cac7146bd None None] MFA Rules not processed for user `018335d2829640e1bda1e3c751ae46f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:09:59.109882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-91ec0c15-677f-4a13-9a5a-664cac7146bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:59.366027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 895/1782] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:09:59 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:09:59.377792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:09:59.379773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:09:59.616861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08879d77-4413-41d0-ac97-b125cca6754d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:09:59.626562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:09:59.627200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:09:59.627481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:09:59.627779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:09:59.652922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:09:59.654374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:09:59.678254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 888/1783] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:09:59 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:00.052858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ec0c15-677f-4a13-9a5a-664cac7146bd None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:00.055138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-91ec0c15-677f-4a13-9a5a-664cac7146bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:00.093426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ec0c15-677f-4a13-9a5a-664cac7146bd None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:00.118312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99805019-0eec-44f8-8646-8236f2030a66 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:00.119936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99805019-0eec-44f8-8646-8236f2030a66 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:00.120417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99805019-0eec-44f8-8646-8236f2030a66 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:00.122505 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99805019-0eec-44f8-8646-8236f2030a66 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:00.166321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:00.178137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:00.179923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:00.290078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:00.325608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99805019-0eec-44f8-8646-8236f2030a66 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:00.331262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99805019-0eec-44f8-8646-8236f2030a66 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:00.389120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 896/1784] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:00.475890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa60a62-f457-4872-8f61-ec7da439746f service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:00.500400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:00.502677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:00.504108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:00.505599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] PATH_INFO: `/v3/users/cdf9054534b6474f88f216794ff39593` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:00.512137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cdf9054534b6474f88f216794ff39593)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:00.518116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:00.690469 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cdf9054534b6474f88f216794ff39593'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:00.714551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cdf9054534b6474f88f216794ff39593'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:00.718015 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cdf9054534b6474f88f216794ff39593'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:00.719719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cdf9054534b6474f88f216794ff39593'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:00.743592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cdf9054534b6474f88f216794ff39593'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:00.820579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 889/1785] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:00 2022] DELETE /identity/v3/users/cdf9054534b6474f88f216794ff39593 => generated 0 bytes in 640 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:00.822353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c89885-a5a3-4bc5-b1fa-71f45b3b61bb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:00.822656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c89885-a5a3-4bc5-b1fa-71f45b3b61bb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:00.822910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c89885-a5a3-4bc5-b1fa-71f45b3b61bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:00.884924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a2c89885-a5a3-4bc5-b1fa-71f45b3b61bb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:00.886666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a2c89885-a5a3-4bc5-b1fa-71f45b3b61bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:01.073506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 897/1786] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:01.212637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:01.214108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:01.461974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2409f0e-f5cf-4476-872f-663ada6a6ed5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:01.462467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2409f0e-f5cf-4476-872f-663ada6a6ed5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:01.462953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2409f0e-f5cf-4476-872f-663ada6a6ed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:01.519822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed8cdee-5ce6-45a3-bf02-6c7eda79b5b7 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:01.545477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff702ef-d593-46a2-89b4-b197c0906b77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:01.546368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff702ef-d593-46a2-89b4-b197c0906b77 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:01.547474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff702ef-d593-46a2-89b4-b197c0906b77 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:01.547985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff702ef-d593-46a2-89b4-b197c0906b77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:01.551592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-b2409f0e-f5cf-4476-872f-663ada6a6ed5 None None] MFA Rules not processed for user `00374d3d3bf649749781792273e35e3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:01.553449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b2409f0e-f5cf-4476-872f-663ada6a6ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:01.589454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff702ef-d593-46a2-89b4-b197c0906b77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:01.596719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff702ef-d593-46a2-89b4-b197c0906b77 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:01.636957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 890/1787] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:01.638544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80415a7-50a0-4d40-af44-1f0c96f774c6 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:01.646611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80415a7-50a0-4d40-af44-1f0c96f774c6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:01.647927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80415a7-50a0-4d40-af44-1f0c96f774c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:01.705876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 898/1788] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:01 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:01.762112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c80415a7-50a0-4d40-af44-1f0c96f774c6 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:01.763748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c80415a7-50a0-4d40-af44-1f0c96f774c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:01.844306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:01.987767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 891/1789] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:10:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 350 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:02.203995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2409f0e-f5cf-4476-872f-663ada6a6ed5 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:02.208260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b2409f0e-f5cf-4476-872f-663ada6a6ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:02.316861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:02.381588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2409f0e-f5cf-4476-872f-663ada6a6ed5 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:02.400923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772744b6-97c5-47aa-8ab8-8ff2f504cc26 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:02.401564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772744b6-97c5-47aa-8ab8-8ff2f504cc26 service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:02.401842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772744b6-97c5-47aa-8ab8-8ff2f504cc26 service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:02.402117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772744b6-97c5-47aa-8ab8-8ff2f504cc26 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:02.449492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772744b6-97c5-47aa-8ab8-8ff2f504cc26 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:02.451295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772744b6-97c5-47aa-8ab8-8ff2f504cc26 service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:02.473594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 899/1790] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:10:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:10:03.199172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80415a7-50a0-4d40-af44-1f0c96f774c6 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:03.200781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c80415a7-50a0-4d40-af44-1f0c96f774c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:03.243227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80415a7-50a0-4d40-af44-1f0c96f774c6 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:03.249064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:03.254089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:03.254473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:03.254821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] PATH_INFO: `/v3/projects/787051300d714b628ecbe666880d0854` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:03.256268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] RBAC: Authorizing `identity:delete_project(project_id=787051300d714b628ecbe666880d0854)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:03.258424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:03.390211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '787051300d714b628ecbe666880d0854', 'request_id': 'req-3428dc48-c98c-45fe-bd01-f85d6c780f03'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.406694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '787051300d714b628ecbe666880d0854', 'request_id': 'req-3428dc48-c98c-45fe-bd01-f85d6c780f03'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.418581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '787051300d714b628ecbe666880d0854', 'request_id': 'req-3428dc48-c98c-45fe-bd01-f85d6c780f03'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.436211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] The token cache is being invalidate because project 787051300d714b628ecbe666880d0854 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:03.436835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.438686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.440978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 892/1791] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:03 2022] DELETE /identity/v3/projects/787051300d714b628ecbe666880d0854 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:03.444690 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772744b6-97c5-47aa-8ab8-8ff2f504cc26 service swift] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:03.466876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772744b6-97c5-47aa-8ab8-8ff2f504cc26 service swift] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:03.548420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:03.550425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:03.550866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:03.551433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] PATH_INFO: `/v3/users/77fcdddd580e462e9c85d0f15ad2044d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:03.553620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] RBAC: Authorizing `identity:delete_user(user_id=77fcdddd580e462e9c85d0f15ad2044d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:03.558683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:03.652655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77fcdddd580e462e9c85d0f15ad2044d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.677624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77fcdddd580e462e9c85d0f15ad2044d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.678256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77fcdddd580e462e9c85d0f15ad2044d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.679071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77fcdddd580e462e9c85d0f15ad2044d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.696898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-59d6ccd8-405f-455c-87ee-55c814c15ffb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77fcdddd580e462e9c85d0f15ad2044d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:03.735041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 900/1792] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:03 2022] DELETE /identity/v3/users/77fcdddd580e462e9c85d0f15ad2044d => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:04.452659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:04.455063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:04.497814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3428dc48-c98c-45fe-bd01-f85d6c780f03 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:04.671731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:04.673085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:04.673538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:04.673848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] PATH_INFO: `/v3/projects/9bf8662b5a4041b0a1c87d34ac76b43f` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:04.675003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9bf8662b5a4041b0a1c87d34ac76b43f)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:04.677125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:04.741048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:04.744165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9bf8662b5a4041b0a1c87d34ac76b43f', 'request_id': 'req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:04.756498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9bf8662b5a4041b0a1c87d34ac76b43f', 'request_id': 'req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:04.766680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9bf8662b5a4041b0a1c87d34ac76b43f', 'request_id': 'req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:04.780273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] The token cache is being invalidate because project 9bf8662b5a4041b0a1c87d34ac76b43f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:04.780570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:04.782710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-60cfcd9a-ddc3-4522-9c63-cdb8bb844dd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:04.785221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 893/1793] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:04 2022] DELETE /identity/v3/projects/9bf8662b5a4041b0a1c87d34ac76b43f => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:04.791656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918270ed-7970-4c9e-bf94-b6062f56efbb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:04.792551 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918270ed-7970-4c9e-bf94-b6062f56efbb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:04.793430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918270ed-7970-4c9e-bf94-b6062f56efbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:04.819533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-918270ed-7970-4c9e-bf94-b6062f56efbb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:04.820964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-918270ed-7970-4c9e-bf94-b6062f56efbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:04.874578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:04.896383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 901/1794] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:10:04.903871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7aca42-a052-4503-88be-61a0c0bb2f9c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:04.904176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7aca42-a052-4503-88be-61a0c0bb2f9c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:04.904449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7aca42-a052-4503-88be-61a0c0bb2f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:04.926522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-5b7aca42-a052-4503-88be-61a0c0bb2f9c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:04.928035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5b7aca42-a052-4503-88be-61a0c0bb2f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:04.939734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 894/1795] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:10:04.941409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918270ed-7970-4c9e-bf94-b6062f56efbb None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:04.951590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918270ed-7970-4c9e-bf94-b6062f56efbb None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:04.957750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:04.958374 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:04.958826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:04.959119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:04.959801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:04.960108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:04.961759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:04.978403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 902/1796] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:04.979987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7aca42-a052-4503-88be-61a0c0bb2f9c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:04.991995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7aca42-a052-4503-88be-61a0c0bb2f9c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:05.024647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd6513e-ee67-4c56-a235-2f1ab282aacb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:05.025780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd6513e-ee67-4c56-a235-2f1ab282aacb admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.026290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd6513e-ee67-4c56-a235-2f1ab282aacb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.026471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd6513e-ee67-4c56-a235-2f1ab282aacb admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.027131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd6513e-ee67-4c56-a235-2f1ab282aacb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:05.029656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd6513e-ee67-4c56-a235-2f1ab282aacb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:05.068784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 895/1797] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:04 2022] POST /identity/v3/projects => generated 381 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:05.076781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:05.088460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b01a778-805d-4138-bc71-6682bcb1e2bb admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:05.156714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2930311d-707b-4415-b637-8e5260586ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:05.157341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2930311d-707b-4415-b637-8e5260586ebb admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.157627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2930311d-707b-4415-b637-8e5260586ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.158020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2930311d-707b-4415-b637-8e5260586ebb admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.158680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2930311d-707b-4415-b637-8e5260586ebb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:05.160724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2930311d-707b-4415-b637-8e5260586ebb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:05.199295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 903/1798] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:05 2022] POST /identity/v3/users => generated 291 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:05.200549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd6513e-ee67-4c56-a235-2f1ab282aacb admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:05.216171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd6513e-ee67-4c56-a235-2f1ab282aacb admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:05.221236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba9df9f-9e85-4951-aa95-1a4a9e9f389e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:05.221870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba9df9f-9e85-4951-aa95-1a4a9e9f389e admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.222226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba9df9f-9e85-4951-aa95-1a4a9e9f389e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.222508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba9df9f-9e85-4951-aa95-1a4a9e9f389e admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.223173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba9df9f-9e85-4951-aa95-1a4a9e9f389e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:05.224827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba9df9f-9e85-4951-aa95-1a4a9e9f389e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:05.238033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 896/1799] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:05 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:10:05.241726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2930311d-707b-4415-b637-8e5260586ebb admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:05.260277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2930311d-707b-4415-b637-8e5260586ebb admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:05.268669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98592af0-8816-4b6a-8146-883055826fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:05.270011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98592af0-8816-4b6a-8146-883055826fe9 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.270483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98592af0-8816-4b6a-8146-883055826fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.271957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98592af0-8816-4b6a-8146-883055826fe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.285861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98592af0-8816-4b6a-8146-883055826fe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:05.287597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98592af0-8816-4b6a-8146-883055826fe9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:05.326958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 904/1800] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:05 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:05.337017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba9df9f-9e85-4951-aa95-1a4a9e9f389e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:05.349443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba9df9f-9e85-4951-aa95-1a4a9e9f389e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:05.359565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3051d131-ffb8-4f5c-9b1b-9b3f282884e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:05.361673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3051d131-ffb8-4f5c-9b1b-9b3f282884e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.361673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3051d131-ffb8-4f5c-9b1b-9b3f282884e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.361673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3051d131-ffb8-4f5c-9b1b-9b3f282884e5 admin admin] PATH_INFO: `/v3/projects/fe4e5a5788094d67a0697b9990f23495/users/1d7070238deb4bb8a486b452bdd9e066/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.380626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3051d131-ffb8-4f5c-9b1b-9b3f282884e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe4e5a5788094d67a0697b9990f23495, user_id=1d7070238deb4bb8a486b452bdd9e066, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:05.383952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3051d131-ffb8-4f5c-9b1b-9b3f282884e5 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:05.408821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 897/1801] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:05 2022] PUT /identity/v3/projects/fe4e5a5788094d67a0697b9990f23495/users/1d7070238deb4bb8a486b452bdd9e066/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:05.423504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f725c5-5fbf-4472-acee-b312b6fa6fff None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.423869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f725c5-5fbf-4472-acee-b312b6fa6fff None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.424160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f725c5-5fbf-4472-acee-b312b6fa6fff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.476190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a7f725c5-5fbf-4472-acee-b312b6fa6fff None None] MFA Rules not processed for user `1d7070238deb4bb8a486b452bdd9e066`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:05.477474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a7f725c5-5fbf-4472-acee-b312b6fa6fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:05.561694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 905/1802] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:05.571640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3051d131-ffb8-4f5c-9b1b-9b3f282884e5 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:05.662459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3051d131-ffb8-4f5c-9b1b-9b3f282884e5 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:05.667924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e428bd-54e7-4f83-bcdd-92332ed2ec37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:05.668797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e428bd-54e7-4f83-bcdd-92332ed2ec37 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.668900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e428bd-54e7-4f83-bcdd-92332ed2ec37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.669121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e428bd-54e7-4f83-bcdd-92332ed2ec37 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.669963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e428bd-54e7-4f83-bcdd-92332ed2ec37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:05.672700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e428bd-54e7-4f83-bcdd-92332ed2ec37 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:05.752759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 898/1803] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:05 2022] POST /identity/v3/projects => generated 381 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:05.756850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f725c5-5fbf-4472-acee-b312b6fa6fff None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:05.775061 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f725c5-5fbf-4472-acee-b312b6fa6fff None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:05.885217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:05.904096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ce7cbb-601c-498c-b06f-feb634cd2146 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:05.905120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ce7cbb-601c-498c-b06f-feb634cd2146 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.905765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ce7cbb-601c-498c-b06f-feb634cd2146 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.906801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ce7cbb-601c-498c-b06f-feb634cd2146 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.907525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ce7cbb-601c-498c-b06f-feb634cd2146 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:05.909550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ce7cbb-601c-498c-b06f-feb634cd2146 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:05.936987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 906/1804] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:05 2022] POST /identity/v3/users => generated 297 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:05.941787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e428bd-54e7-4f83-bcdd-92332ed2ec37 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:05.955701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e428bd-54e7-4f83-bcdd-92332ed2ec37 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:05.969033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa55837d-ecfa-4041-97ca-fd24644153a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:05.969946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa55837d-ecfa-4041-97ca-fd24644153a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:05.970307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa55837d-ecfa-4041-97ca-fd24644153a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:05.970678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa55837d-ecfa-4041-97ca-fd24644153a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:05.971426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa55837d-ecfa-4041-97ca-fd24644153a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:05.973222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa55837d-ecfa-4041-97ca-fd24644153a8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:05.988692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 899/1805] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:05 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:05.994099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ce7cbb-601c-498c-b06f-feb634cd2146 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:06.009125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ce7cbb-601c-498c-b06f-feb634cd2146 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:06.014009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295d38a8-6a50-4039-a567-1f42ccf0cd07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:06.014692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295d38a8-6a50-4039-a567-1f42ccf0cd07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:06.014986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295d38a8-6a50-4039-a567-1f42ccf0cd07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:06.015283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295d38a8-6a50-4039-a567-1f42ccf0cd07 admin admin] PATH_INFO: `/v3/projects/a540ff5f8aef42f6987da6c67ed36cca/users/5add721e656f488e942b8855f1c36840/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:06.038468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295d38a8-6a50-4039-a567-1f42ccf0cd07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a540ff5f8aef42f6987da6c67ed36cca, user_id=5add721e656f488e942b8855f1c36840, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:06.041588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295d38a8-6a50-4039-a567-1f42ccf0cd07 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:06.070725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 907/1806] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:05 2022] PUT /identity/v3/projects/a540ff5f8aef42f6987da6c67ed36cca/users/5add721e656f488e942b8855f1c36840/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:06.074864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa55837d-ecfa-4041-97ca-fd24644153a8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:06.092964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa55837d-ecfa-4041-97ca-fd24644153a8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:06.183045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:06.203668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b150615-87d1-43ab-9467-5b905673c8e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:06.205346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b150615-87d1-43ab-9467-5b905673c8e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:06.206200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b150615-87d1-43ab-9467-5b905673c8e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:06.206761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b150615-87d1-43ab-9467-5b905673c8e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:06.209359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b150615-87d1-43ab-9467-5b905673c8e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:06.211608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b150615-87d1-43ab-9467-5b905673c8e4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:06.227731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 900/1807] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:06 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:06.230419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295d38a8-6a50-4039-a567-1f42ccf0cd07 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:06.242230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295d38a8-6a50-4039-a567-1f42ccf0cd07 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:06.247785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:06.248396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:06.248721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:06.248946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] PATH_INFO: `/v3/projects/a540ff5f8aef42f6987da6c67ed36cca/users/5add721e656f488e942b8855f1c36840/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:06.251669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a540ff5f8aef42f6987da6c67ed36cca, user_id=5add721e656f488e942b8855f1c36840, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:06.254442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:06.267520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 908/1808] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:06 2022] PUT /identity/v3/projects/a540ff5f8aef42f6987da6c67ed36cca/users/5add721e656f488e942b8855f1c36840/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:06.278676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c5924-0b2a-49df-8b4e-42bfe0a1874d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:06.279489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c5924-0b2a-49df-8b4e-42bfe0a1874d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:06.280204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820c5924-0b2a-49df-8b4e-42bfe0a1874d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:06.349136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-820c5924-0b2a-49df-8b4e-42bfe0a1874d None None] MFA Rules not processed for user `5add721e656f488e942b8855f1c36840`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:06.350386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-820c5924-0b2a-49df-8b4e-42bfe0a1874d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:06.586569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 901/1809] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:06 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:06.603424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:06.613725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:06.746763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d262bb3-a190-4dd5-a4b0-5c647033e331 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:06.754255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d8aaa3-f8aa-42cc-9f15-7f04109c825e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:06.754887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8aaa3-f8aa-42cc-9f15-7f04109c825e service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:06.755165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8aaa3-f8aa-42cc-9f15-7f04109c825e service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:06.755436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d8aaa3-f8aa-42cc-9f15-7f04109c825e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:06.769135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d8aaa3-f8aa-42cc-9f15-7f04109c825e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:06.780698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d8aaa3-f8aa-42cc-9f15-7f04109c825e service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:06.800463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 909/1810] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:06 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:07.361100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820c5924-0b2a-49df-8b4e-42bfe0a1874d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:07.514789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820c5924-0b2a-49df-8b4e-42bfe0a1874d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:07.520880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:07.521545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:07.522038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:07.522322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] PATH_INFO: `/v3/users/1d7070238deb4bb8a486b452bdd9e066` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:07.523555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d7070238deb4bb8a486b452bdd9e066)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:07.526198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:07.626985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d7070238deb4bb8a486b452bdd9e066'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:07.648222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d7070238deb4bb8a486b452bdd9e066'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:07.649998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d7070238deb4bb8a486b452bdd9e066'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:07.653539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d7070238deb4bb8a486b452bdd9e066'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:07.680770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d7070238deb4bb8a486b452bdd9e066'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:07.692086 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:07.738984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 902/1811] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:07 2022] DELETE /identity/v3/users/1d7070238deb4bb8a486b452bdd9e066 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:07.743194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a79586-0a19-4412-9c33-9024a60cdbdd None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:07.743750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a79586-0a19-4412-9c33-9024a60cdbdd None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:07.743848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a79586-0a19-4412-9c33-9024a60cdbdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:07.780239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-14a79586-0a19-4412-9c33-9024a60cdbdd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:07.781715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-14a79586-0a19-4412-9c33-9024a60cdbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:07.990754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 910/1812] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:08.006668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:08.009163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:08.130494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8647818-0c4f-4de9-bae9-12dc0d7064cb admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:08.134315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3cf1f-148c-470e-9a3d-8bce502b23c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:08.134914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3cf1f-148c-470e-9a3d-8bce502b23c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:08.135199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3cf1f-148c-470e-9a3d-8bce502b23c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:08.135499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3cf1f-148c-470e-9a3d-8bce502b23c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:08.163958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f3cf1f-148c-470e-9a3d-8bce502b23c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:08.163958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f3cf1f-148c-470e-9a3d-8bce502b23c7 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:08.190170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 903/1813] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:08.621717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a79586-0a19-4412-9c33-9024a60cdbdd None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:08.623309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-14a79586-0a19-4412-9c33-9024a60cdbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:08.672099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a79586-0a19-4412-9c33-9024a60cdbdd None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:08.678060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:08.678724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:08.679005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:08.679281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] PATH_INFO: `/v3/projects/fe4e5a5788094d67a0697b9990f23495` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:08.680570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe4e5a5788094d67a0697b9990f23495)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:08.684048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:08.808273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:08.864769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe4e5a5788094d67a0697b9990f23495', 'request_id': 'req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:08.900221 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe4e5a5788094d67a0697b9990f23495', 'request_id': 'req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:08.911269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe4e5a5788094d67a0697b9990f23495', 'request_id': 'req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:08.942960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] The token cache is being invalidate because project fe4e5a5788094d67a0697b9990f23495 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:08.943478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:08.944442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:08.951341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 911/1814] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:08 2022] DELETE /identity/v3/projects/fe4e5a5788094d67a0697b9990f23495 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:08.952548 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3cf1f-148c-470e-9a3d-8bce502b23c7 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:08.977296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3cf1f-148c-470e-9a3d-8bce502b23c7 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:09.095099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:09.096829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:09.150143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:09.152056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:09.152503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:09.152928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] PATH_INFO: `/v3/users/5add721e656f488e942b8855f1c36840` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:09.154305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] RBAC: Authorizing `identity:delete_user(user_id=5add721e656f488e942b8855f1c36840)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:09.156833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:09.221184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef1dc4a-3f4c-4f90-a03b-1082cde753b3 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:09.229735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:09.230784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:09.231223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:09.231652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] PATH_INFO: `/v3/users/426ee1c4c0cb44cc8b33cfe29e9dfaa8` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:09.233680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=426ee1c4c0cb44cc8b33cfe29e9dfaa8)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:09.237972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:09.279385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5add721e656f488e942b8855f1c36840'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.293350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5add721e656f488e942b8855f1c36840'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.295040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5add721e656f488e942b8855f1c36840'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.296587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5add721e656f488e942b8855f1c36840'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.319542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aefa294f-417b-40b0-9b5c-419f221b089e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5add721e656f488e942b8855f1c36840'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.356457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '426ee1c4c0cb44cc8b33cfe29e9dfaa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.362454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 904/1815] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:08 2022] DELETE /identity/v3/users/5add721e656f488e942b8855f1c36840 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:09.369755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '426ee1c4c0cb44cc8b33cfe29e9dfaa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.370837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '426ee1c4c0cb44cc8b33cfe29e9dfaa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.372058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '426ee1c4c0cb44cc8b33cfe29e9dfaa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.402614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '426ee1c4c0cb44cc8b33cfe29e9dfaa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.433570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 912/1816] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:09 2022] DELETE /identity/v3/users/426ee1c4c0cb44cc8b33cfe29e9dfaa8 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:09.438460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce734e5a-f4ae-4959-8093-be2178782cec None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:09.438722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce734e5a-f4ae-4959-8093-be2178782cec None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:09.439047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce734e5a-f4ae-4959-8093-be2178782cec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:09.464069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:09.479498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ce734e5a-f4ae-4959-8093-be2178782cec None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:09.483296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ce734e5a-f4ae-4959-8093-be2178782cec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:09.690809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 905/1817] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:09.697648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:09.709153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:09.830215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce734e5a-f4ae-4959-8093-be2178782cec None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:09.831997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ce734e5a-f4ae-4959-8093-be2178782cec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:09.861345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:09.868412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce734e5a-f4ae-4959-8093-be2178782cec None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:09.877096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:09.882387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:09.882387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:09.882387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] PATH_INFO: `/v3/projects/a540ff5f8aef42f6987da6c67ed36cca` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:09.883334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=a540ff5f8aef42f6987da6c67ed36cca)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:09.887957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:09.893200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a581317-68b1-45d2-ad61-741fe51c27a6 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:09.903005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a657369-56e0-47b3-9e89-f985692ec282 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:09.904145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a657369-56e0-47b3-9e89-f985692ec282 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:09.913871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a657369-56e0-47b3-9e89-f985692ec282 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:09.914605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a657369-56e0-47b3-9e89-f985692ec282 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:09.928281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a657369-56e0-47b3-9e89-f985692ec282 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:09.930362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a657369-56e0-47b3-9e89-f985692ec282 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:09.955778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a540ff5f8aef42f6987da6c67ed36cca', 'request_id': 'req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.956457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 913/1818] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:09.966691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a540ff5f8aef42f6987da6c67ed36cca', 'request_id': 'req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.977170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a540ff5f8aef42f6987da6c67ed36cca', 'request_id': 'req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:09.998852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] The token cache is being invalidate because project a540ff5f8aef42f6987da6c67ed36cca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:10.000926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:10.002498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6be5adc8-0b0d-4839-9415-ac8f3709b3fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:10.005021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 906/1819] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:09 2022] DELETE /identity/v3/projects/a540ff5f8aef42f6987da6c67ed36cca => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:10.011141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129151d4-6fc2-4667-840f-67917fbfceb9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:10.011522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129151d4-6fc2-4667-840f-67917fbfceb9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:10.011900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129151d4-6fc2-4667-840f-67917fbfceb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:10.059070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-129151d4-6fc2-4667-840f-67917fbfceb9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:10.066353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-129151d4-6fc2-4667-840f-67917fbfceb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:10.252935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 914/1820] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:10.260799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf56a3b8-b3bf-4698-bb79-9369711f2206 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:10.261093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf56a3b8-b3bf-4698-bb79-9369711f2206 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:10.261818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf56a3b8-b3bf-4698-bb79-9369711f2206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:10.288821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-cf56a3b8-b3bf-4698-bb79-9369711f2206 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:10.290188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-cf56a3b8-b3bf-4698-bb79-9369711f2206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:10.319677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 907/1821] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:10.333529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129151d4-6fc2-4667-840f-67917fbfceb9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:10.342113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129151d4-6fc2-4667-840f-67917fbfceb9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:10.373467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:10.376964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:10.378763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:10.379181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:10.381385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:10.381815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:10.387563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:10.444625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 915/1822] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:10.446925 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf56a3b8-b3bf-4698-bb79-9369711f2206 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:10.459399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf56a3b8-b3bf-4698-bb79-9369711f2206 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:10.486864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:10.487931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:10.490504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:10.490938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:10.491934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:10.495619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:10.574651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 908/1823] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:10 2022] POST /identity/v3/projects => generated 385 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:10.576713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:10.607118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16197cd-b809-487f-806d-5215b27fb4dd admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:10.627838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:10.629178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:10.788524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7961ca9f-3f8b-4496-afa8-62372b294909 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:10.811369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961ca9f-3f8b-4496-afa8-62372b294909 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:10.811863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961ca9f-3f8b-4496-afa8-62372b294909 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:10.812123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961ca9f-3f8b-4496-afa8-62372b294909 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:10.813999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7961ca9f-3f8b-4496-afa8-62372b294909 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:10.820590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7961ca9f-3f8b-4496-afa8-62372b294909 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:10.854146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 916/1824] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:10 2022] POST /identity/v3/users => generated 293 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:10:10.859470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7961ca9f-3f8b-4496-afa8-62372b294909 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:10.882976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7961ca9f-3f8b-4496-afa8-62372b294909 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:10.910559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824db606-5175-4fb1-8b8a-ece0c7409f02 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:10.918535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:10.919201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:10.919486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:10.919785 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] PATH_INFO: `/v3/projects/26eec6155e2d45f78968cd98e9b4226c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:10.922262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26eec6155e2d45f78968cd98e9b4226c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:10.923709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaf975c-4bcd-4fd4-bd75-80390a029d0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:10.924419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaf975c-4bcd-4fd4-bd75-80390a029d0d admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:10.924633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaf975c-4bcd-4fd4-bd75-80390a029d0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:10.924905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaf975c-4bcd-4fd4-bd75-80390a029d0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:10.926239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaf975c-4bcd-4fd4-bd75-80390a029d0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:10.928070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:10.930960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaf975c-4bcd-4fd4-bd75-80390a029d0d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:10.933142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:10.959468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 917/1825] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:10 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:10.960524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaf975c-4bcd-4fd4-bd75-80390a029d0d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:11.006369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaf975c-4bcd-4fd4-bd75-80390a029d0d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:11.024609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc30babc-12b2-4575-9ea5-54b717d1610c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:11.025235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc30babc-12b2-4575-9ea5-54b717d1610c admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:11.025718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc30babc-12b2-4575-9ea5-54b717d1610c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:11.026063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc30babc-12b2-4575-9ea5-54b717d1610c admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:11.026724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc30babc-12b2-4575-9ea5-54b717d1610c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:11.028386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc30babc-12b2-4575-9ea5-54b717d1610c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:11.044910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 918/1826] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:10 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:11.045840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc30babc-12b2-4575-9ea5-54b717d1610c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:11.057005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc30babc-12b2-4575-9ea5-54b717d1610c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:11.134382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26eec6155e2d45f78968cd98e9b4226c', 'request_id': 'req-9e09d61e-a664-4e43-b112-57d8092b2879'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.155215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26eec6155e2d45f78968cd98e9b4226c', 'request_id': 'req-9e09d61e-a664-4e43-b112-57d8092b2879'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.176100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26eec6155e2d45f78968cd98e9b4226c', 'request_id': 'req-9e09d61e-a664-4e43-b112-57d8092b2879'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.184109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:11.210895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] The token cache is being invalidate because project 26eec6155e2d45f78968cd98e9b4226c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:11.211261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.211988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.217837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 909/1827] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:10 2022] DELETE /identity/v3/projects/26eec6155e2d45f78968cd98e9b4226c => generated 0 bytes in 587 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:11.219535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:11.221381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:11.239115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cefaefb-ac50-44cb-92bb-486581fe5957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:11.239853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cefaefb-ac50-44cb-92bb-486581fe5957 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:11.240125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cefaefb-ac50-44cb-92bb-486581fe5957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:11.240409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cefaefb-ac50-44cb-92bb-486581fe5957 admin admin] PATH_INFO: `/v3/projects/1705225e489641e3a8d34a4d0fa49812/users/d6e2c4400a564ed6a68493621e248850/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:11.268682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cefaefb-ac50-44cb-92bb-486581fe5957 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1705225e489641e3a8d34a4d0fa49812, user_id=d6e2c4400a564ed6a68493621e248850, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:11.273541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cefaefb-ac50-44cb-92bb-486581fe5957 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:11.296921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e09d61e-a664-4e43-b112-57d8092b2879 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:11.323440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 919/1828] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:11 2022] PUT /identity/v3/projects/1705225e489641e3a8d34a4d0fa49812/users/d6e2c4400a564ed6a68493621e248850/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:11.335007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f60226a-2005-491b-9e64-cb11fb0a717e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:11.335355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f60226a-2005-491b-9e64-cb11fb0a717e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:11.335623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f60226a-2005-491b-9e64-cb11fb0a717e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:11.496446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1f60226a-2005-491b-9e64-cb11fb0a717e None None] MFA Rules not processed for user `d6e2c4400a564ed6a68493621e248850`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:11.504115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1f60226a-2005-491b-9e64-cb11fb0a717e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:11.506526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:11.507418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:11.507869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:11.508337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] PATH_INFO: `/v3/users/018335d2829640e1bda1e3c751ae46f2` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:11.516842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] RBAC: Authorizing `identity:delete_user(user_id=018335d2829640e1bda1e3c751ae46f2)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:11.518994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:11.686869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '018335d2829640e1bda1e3c751ae46f2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.705897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 920/1829] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:11 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 372 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:11.708760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2522d4c7-88c1-4756-894c-b6f1a86f2eb9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:11.713839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2522d4c7-88c1-4756-894c-b6f1a86f2eb9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:11.714471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2522d4c7-88c1-4756-894c-b6f1a86f2eb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:11.730631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '018335d2829640e1bda1e3c751ae46f2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.731410 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '018335d2829640e1bda1e3c751ae46f2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.739565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '018335d2829640e1bda1e3c751ae46f2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.763583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2522d4c7-88c1-4756-894c-b6f1a86f2eb9 None None] MFA Rules not processed for user `922012eacf244f9480c180c2eb9eaf28`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:11.767515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2522d4c7-88c1-4756-894c-b6f1a86f2eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:11.780566 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '018335d2829640e1bda1e3c751ae46f2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:11.818361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 910/1830] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:11 2022] DELETE /identity/v3/users/018335d2829640e1bda1e3c751ae46f2 => generated 0 bytes in 600 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:11.819485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:11.831027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:11.889743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 921/1831] 10.222.0.52 () {58 vars in 1015 bytes} [Thu Sep 15 14:10:11 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:11.894833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2522d4c7-88c1-4756-894c-b6f1a86f2eb9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:11.898163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2522d4c7-88c1-4756-894c-b6f1a86f2eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:11.915049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:12.094049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab9f22-2e6c-4f49-97a8-086cc7ca652e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:12.105694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:12.106421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:12.106711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:12.107381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] PATH_INFO: `/v3/users/238ebdc5126348e19b4cea1677dba8f3` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:12.112049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=238ebdc5126348e19b4cea1677dba8f3)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:12.115197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:12.120828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2522d4c7-88c1-4756-894c-b6f1a86f2eb9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:12.154871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:12.247159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '238ebdc5126348e19b4cea1677dba8f3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:12.263147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '238ebdc5126348e19b4cea1677dba8f3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:12.264224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '238ebdc5126348e19b4cea1677dba8f3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:12.265085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '238ebdc5126348e19b4cea1677dba8f3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:12.271448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:12.271448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:12.271448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:12.271448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:12.272664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:12.281868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-67890a71-67cf-4e01-bf98-e90fc180d1d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '238ebdc5126348e19b4cea1677dba8f3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:12.377866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 911/1832] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:11 2022] DELETE /identity/v3/users/238ebdc5126348e19b4cea1677dba8f3 => generated 0 bytes in 559 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:12.383272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121876ff-d7e3-4733-913b-d3eb9f755da3 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:12.383817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121876ff-d7e3-4733-913b-d3eb9f755da3 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:12.384507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121876ff-d7e3-4733-913b-d3eb9f755da3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:12.449602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-121876ff-d7e3-4733-913b-d3eb9f755da3 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:12.451008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-121876ff-d7e3-4733-913b-d3eb9f755da3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:12.457189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:12.458872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:12.470604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 922/1833] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:11 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 578 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:12.471636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:12.473491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:12.662592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d403673-b4dd-43bd-b178-1f858640052d service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:12.669676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 912/1834] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:10:12.670922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121876ff-d7e3-4733-913b-d3eb9f755da3 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:12.674489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-121876ff-d7e3-4733-913b-d3eb9f755da3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:12.782199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:12.829096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:12.831574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:12.831732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:12.831969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:12.835200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:12.842834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121876ff-d7e3-4733-913b-d3eb9f755da3 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:12.856722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:12.858977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:12.867623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:12.868392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:12.869131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:12.869504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:12.878911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 923/1835] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:12.880007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:12.881933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:12.889804 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:12.902302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:12.908228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8fcfee-a79e-4455-a10c-b0207dd837a0 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:12.918934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 913/1836] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:12.920064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:12.921070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:12.924392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:12.924897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:12.925287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:12.925857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] PATH_INFO: `/v3/users/d6e2c4400a564ed6a68493621e248850` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:12.928774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6e2c4400a564ed6a68493621e248850)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:12.935226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:12.960021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229f308a-2120-464e-8914-12195517a9cf service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:12.974164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:12.975486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:12.976029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:12.976405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] PATH_INFO: `/v3/projects/9465e3a9227145e4823542de34777eae` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:12.982194 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9465e3a9227145e4823542de34777eae)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:12.989915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:13.033578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:13.080578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6e2c4400a564ed6a68493621e248850'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.091148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6e2c4400a564ed6a68493621e248850'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.092367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6e2c4400a564ed6a68493621e248850'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.094682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6e2c4400a564ed6a68493621e248850'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.113445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6e2c4400a564ed6a68493621e248850'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.159085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9465e3a9227145e4823542de34777eae', 'request_id': 'req-852e83d6-c14f-469c-9955-9c03e10f6d1e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.165632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 924/1837] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:12 2022] DELETE /identity/v3/users/d6e2c4400a564ed6a68493621e248850 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:13.166849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:13.168920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:13.179459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9465e3a9227145e4823542de34777eae', 'request_id': 'req-852e83d6-c14f-469c-9955-9c03e10f6d1e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.193151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9465e3a9227145e4823542de34777eae', 'request_id': 'req-852e83d6-c14f-469c-9955-9c03e10f6d1e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.251592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] The token cache is being invalidate because project 9465e3a9227145e4823542de34777eae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:13.252306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.256335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-852e83d6-c14f-469c-9955-9c03e10f6d1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:13.262392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 914/1838] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:12 2022] DELETE /identity/v3/projects/9465e3a9227145e4823542de34777eae => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:13.265713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25cbe12-037f-4496-96f9-c946e2e78cee None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:13.265998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25cbe12-037f-4496-96f9-c946e2e78cee None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:13.266269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25cbe12-037f-4496-96f9-c946e2e78cee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:13.317624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e25cbe12-037f-4496-96f9-c946e2e78cee None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:13.318866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e25cbe12-037f-4496-96f9-c946e2e78cee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:13.339992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f3ce6f-c742-4c8e-8a2c-6fc674db8f33 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:13.349823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:13.353913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:13.354764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:13.355778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:13.362496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:13.463404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 915/1839] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:13.467732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fbbbbf-1de0-443a-a094-5c780a2d079e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:13.468120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fbbbbf-1de0-443a-a094-5c780a2d079e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:13.468343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fbbbbf-1de0-443a-a094-5c780a2d079e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:13.524083 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-53fbbbbf-1de0-443a-a094-5c780a2d079e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:13.525350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-53fbbbbf-1de0-443a-a094-5c780a2d079e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:13.548414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 916/1840] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:13.549357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fbbbbf-1de0-443a-a094-5c780a2d079e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:13.552457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-53fbbbbf-1de0-443a-a094-5c780a2d079e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:13.593220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fbbbbf-1de0-443a-a094-5c780a2d079e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:13.600984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd5faa-c609-40fd-a9ee-103aafe814c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:13.601624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd5faa-c609-40fd-a9ee-103aafe814c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:13.601910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd5faa-c609-40fd-a9ee-103aafe814c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:13.602185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd5faa-c609-40fd-a9ee-103aafe814c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:13.632297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd5faa-c609-40fd-a9ee-103aafe814c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:13.633793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd5faa-c609-40fd-a9ee-103aafe814c0 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:13.669628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 917/1841] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:13.681894 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4771f7-d1ca-4db5-96b7-f66695acc226 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:13.682671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4771f7-d1ca-4db5-96b7-f66695acc226 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:13.682783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4771f7-d1ca-4db5-96b7-f66695acc226 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:13.697728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:13.702540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:13.742484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-7f4771f7-d1ca-4db5-96b7-f66695acc226 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:13.743994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7f4771f7-d1ca-4db5-96b7-f66695acc226 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:13.755548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 925/1842] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 589 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:13.763864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 918/1843] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:13.768334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:13.800050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990bb837-d0a5-41f2-8545-3ae38edd5d66 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:13.820941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:13.821877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:13.822412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:13.823091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:13.824062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:13.828200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:13.842858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:13.903817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 926/1844] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:13.908189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4771f7-d1ca-4db5-96b7-f66695acc226 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:13.924340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4771f7-d1ca-4db5-96b7-f66695acc226 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:13.935732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10e5b07-6355-4766-b007-a4c6a6b6fb6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:13.936776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e5b07-6355-4766-b007-a4c6a6b6fb6b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:13.937607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e5b07-6355-4766-b007-a4c6a6b6fb6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:13.938349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e5b07-6355-4766-b007-a4c6a6b6fb6b admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:13.940408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10e5b07-6355-4766-b007-a4c6a6b6fb6b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:13.956712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10e5b07-6355-4766-b007-a4c6a6b6fb6b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:14.025607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 919/1845] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:13 2022] POST /identity/v3/projects => generated 379 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:14.029656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:14.047231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093a2ab3-bd68-4fed-a01e-dbe50f99f998 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:14.163559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:14.288208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426b4649-ecce-4b03-aa2c-9f3802d38bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:14.289091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b4649-ecce-4b03-aa2c-9f3802d38bdf admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:14.289573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b4649-ecce-4b03-aa2c-9f3802d38bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:14.290832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b4649-ecce-4b03-aa2c-9f3802d38bdf admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:14.292761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426b4649-ecce-4b03-aa2c-9f3802d38bdf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:14.302964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426b4649-ecce-4b03-aa2c-9f3802d38bdf admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:14.355947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 927/1846] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:14 2022] POST /identity/v3/users => generated 290 bytes in 327 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:14.357790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10e5b07-6355-4766-b007-a4c6a6b6fb6b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:14.372797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10e5b07-6355-4766-b007-a4c6a6b6fb6b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:14.379202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897b73ff-6db7-4e16-b94c-3fc5e4581ca0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:14.379904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b73ff-6db7-4e16-b94c-3fc5e4581ca0 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:14.380194 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b73ff-6db7-4e16-b94c-3fc5e4581ca0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:14.380484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897b73ff-6db7-4e16-b94c-3fc5e4581ca0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:14.381167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897b73ff-6db7-4e16-b94c-3fc5e4581ca0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:14.382822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897b73ff-6db7-4e16-b94c-3fc5e4581ca0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:14.385316 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:14.424741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 920/1847] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:14 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:14.426458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426b4649-ecce-4b03-aa2c-9f3802d38bdf admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:14.463495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426b4649-ecce-4b03-aa2c-9f3802d38bdf admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:14.474966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc15f2d-5902-4318-bb82-6c53059d1f5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:14.477770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc15f2d-5902-4318-bb82-6c53059d1f5d admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:14.478145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc15f2d-5902-4318-bb82-6c53059d1f5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:14.483585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc15f2d-5902-4318-bb82-6c53059d1f5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:14.485028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc15f2d-5902-4318-bb82-6c53059d1f5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:14.486824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc15f2d-5902-4318-bb82-6c53059d1f5d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:14.529955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 928/1848] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:14 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:14.534472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897b73ff-6db7-4e16-b94c-3fc5e4581ca0 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:14.554681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897b73ff-6db7-4e16-b94c-3fc5e4581ca0 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:14.580573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377e7c00-2f11-46e7-a778-7de723cdf285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:14.581258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e7c00-2f11-46e7-a778-7de723cdf285 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:14.581541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e7c00-2f11-46e7-a778-7de723cdf285 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:14.581958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e7c00-2f11-46e7-a778-7de723cdf285 admin admin] PATH_INFO: `/v3/projects/9d8a20069ce746b4a0e105f515a812f5/users/0d6be58caa1246f7b3819c5ec8d3b8b6/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:14.628406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377e7c00-2f11-46e7-a778-7de723cdf285 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d8a20069ce746b4a0e105f515a812f5, user_id=0d6be58caa1246f7b3819c5ec8d3b8b6, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:14.634330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377e7c00-2f11-46e7-a778-7de723cdf285 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:14.654310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 921/1849] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:14 2022] PUT /identity/v3/projects/9d8a20069ce746b4a0e105f515a812f5/users/0d6be58caa1246f7b3819c5ec8d3b8b6/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:14.665167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0683c1-1e78-4f96-b21c-192b6dc57893 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:14.665452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0683c1-1e78-4f96-b21c-192b6dc57893 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:14.665822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0683c1-1e78-4f96-b21c-192b6dc57893 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:14.776987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9e0683c1-1e78-4f96-b21c-192b6dc57893 None None] MFA Rules not processed for user `0d6be58caa1246f7b3819c5ec8d3b8b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:14.785078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9e0683c1-1e78-4f96-b21c-192b6dc57893 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:15.016235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 929/1850] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:14 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 352 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:15.021733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377e7c00-2f11-46e7-a778-7de723cdf285 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:15.151314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377e7c00-2f11-46e7-a778-7de723cdf285 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:15.161659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:15.163352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.163975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.164470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.165447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:15.168311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:15.222414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 922/1851] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:15 2022] POST /identity/v3/projects => generated 377 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:15.224903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0683c1-1e78-4f96-b21c-192b6dc57893 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:15.244745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0683c1-1e78-4f96-b21c-192b6dc57893 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:15.353311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06a5d97-bac6-4ea1-aecd-46013baa4bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:15.354822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a5d97-bac6-4ea1-aecd-46013baa4bea admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.355297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a5d97-bac6-4ea1-aecd-46013baa4bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.355799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a5d97-bac6-4ea1-aecd-46013baa4bea admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.356643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06a5d97-bac6-4ea1-aecd-46013baa4bea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:15.358835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06a5d97-bac6-4ea1-aecd-46013baa4bea admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:15.388577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:15.390774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 930/1852] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:15 2022] POST /identity/v3/users => generated 289 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:15.392467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:15.397490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06a5d97-bac6-4ea1-aecd-46013baa4bea admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:15.411261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef83c25-419d-4b39-9049-e5f8aaadd9c1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:15.421820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06a5d97-bac6-4ea1-aecd-46013baa4bea admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:15.424411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:15.425248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.425613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.425991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] PATH_INFO: `/v3/projects/1705225e489641e3a8d34a4d0fa49812` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.428072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1705225e489641e3a8d34a4d0fa49812)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:15.431190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:15.447842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7987df28-9730-4445-bf8c-34ef185723fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:15.448608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7987df28-9730-4445-bf8c-34ef185723fb admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.450915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7987df28-9730-4445-bf8c-34ef185723fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.451840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7987df28-9730-4445-bf8c-34ef185723fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.453352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7987df28-9730-4445-bf8c-34ef185723fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:15.460198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7987df28-9730-4445-bf8c-34ef185723fb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:15.499643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 931/1853] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:15 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:15.501091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7987df28-9730-4445-bf8c-34ef185723fb admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:15.520265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7987df28-9730-4445-bf8c-34ef185723fb admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:15.538573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5245cb-b625-4b37-951d-227ce6860ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:15.540302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5245cb-b625-4b37-951d-227ce6860ba4 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.541403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5245cb-b625-4b37-951d-227ce6860ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.542028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5245cb-b625-4b37-951d-227ce6860ba4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.542578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5245cb-b625-4b37-951d-227ce6860ba4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:15.544722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5245cb-b625-4b37-951d-227ce6860ba4 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:15.564426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1705225e489641e3a8d34a4d0fa49812', 'request_id': 'req-37bc3328-6c74-4a6a-81be-881edf326728'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:15.571443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 932/1854] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:15 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:10:15.572925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33100aff-8601-4370-a8eb-e77538655ebf None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.573486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33100aff-8601-4370-a8eb-e77538655ebf None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.573749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33100aff-8601-4370-a8eb-e77538655ebf None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.575544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 933/1855] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:10:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:10:15.576627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33100aff-8601-4370-a8eb-e77538655ebf None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:15.581274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1705225e489641e3a8d34a4d0fa49812', 'request_id': 'req-37bc3328-6c74-4a6a-81be-881edf326728'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:15.588461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1705225e489641e3a8d34a4d0fa49812', 'request_id': 'req-37bc3328-6c74-4a6a-81be-881edf326728'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:15.603995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] The token cache is being invalidate because project 1705225e489641e3a8d34a4d0fa49812 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:15.604438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:15.605312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-37bc3328-6c74-4a6a-81be-881edf326728 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:15.610614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 923/1856] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:15 2022] DELETE /identity/v3/projects/1705225e489641e3a8d34a4d0fa49812 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:15.612405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a21c885-f350-44bd-a275-63d4c2760f27 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.612733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a21c885-f350-44bd-a275-63d4c2760f27 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.613104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a21c885-f350-44bd-a275-63d4c2760f27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.651628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2a21c885-f350-44bd-a275-63d4c2760f27 None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:15.651628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2a21c885-f350-44bd-a275-63d4c2760f27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:15.659499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:15.711602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:15.747288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33100aff-8601-4370-a8eb-e77538655ebf None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:15.758711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:15.759087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.759198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.759422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] PATH_INFO: `/v3/projects/f7e31117dc6042c19036459c82f15316/users/742d680f400e4f69a7768aad9da289c5/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.778882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 924/1857] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:10:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:15.780001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98767924-b399-4bdd-bab0-3b14b5471ccd None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.780285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98767924-b399-4bdd-bab0-3b14b5471ccd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.780564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98767924-b399-4bdd-bab0-3b14b5471ccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.794244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7e31117dc6042c19036459c82f15316, user_id=742d680f400e4f69a7768aad9da289c5, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:15.797393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:15.813605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-98767924-b399-4bdd-bab0-3b14b5471ccd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:15.815278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-98767924-b399-4bdd-bab0-3b14b5471ccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:15.825756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 934/1858] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:15 2022] PUT /identity/v3/projects/f7e31117dc6042c19036459c82f15316/users/742d680f400e4f69a7768aad9da289c5/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:15.826986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:15.831338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:15.939906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 925/1859] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:15.939906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0478727-ecef-4e4a-9477-845f0cd1983f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.939906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0478727-ecef-4e4a-9477-845f0cd1983f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.939906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0478727-ecef-4e4a-9477-845f0cd1983f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.946910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65473fc9-2c56-4496-ba95-2fbcfe2d3708 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:15.953952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:15.954708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:15.955090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:15.955483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:15.966870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:15.969352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:15.982337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 935/1860] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:15.983137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:15.985827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.001458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169a4e71-c60b-4c0c-a6bc-860936516231 service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.006568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.006705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e0478727-ecef-4e4a-9477-845f0cd1983f None None] MFA Rules not processed for user `742d680f400e4f69a7768aad9da289c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:16.007402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.007816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.007949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e0478727-ecef-4e4a-9477-845f0cd1983f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.008274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] PATH_INFO: `/v3/projects/2372f682fec44246b5845703eeab5df4` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.009498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2372f682fec44246b5845703eeab5df4)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.011741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.118041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2372f682fec44246b5845703eeab5df4', 'request_id': 'req-3852fd59-bc11-4e67-a8a3-c9863537de77'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.132514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2372f682fec44246b5845703eeab5df4', 'request_id': 'req-3852fd59-bc11-4e67-a8a3-c9863537de77'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.135494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 926/1861] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:16.141610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56ca9a3-7398-4c38-837d-b5d1ec8df1c1 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.142162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56ca9a3-7398-4c38-837d-b5d1ec8df1c1 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.144784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56ca9a3-7398-4c38-837d-b5d1ec8df1c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.149046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2372f682fec44246b5845703eeab5df4', 'request_id': 'req-3852fd59-bc11-4e67-a8a3-c9863537de77'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.168950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] The token cache is being invalidate because project 2372f682fec44246b5845703eeab5df4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:16.169559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.170619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.176210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 936/1862] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:15 2022] DELETE /identity/v3/projects/2372f682fec44246b5845703eeab5df4 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:16.176210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.176449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d56ca9a3-7398-4c38-837d-b5d1ec8df1c1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:16.177592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d56ca9a3-7398-4c38-837d-b5d1ec8df1c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.178631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.197002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:16.288962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 927/1863] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:16.290435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56ca9a3-7398-4c38-837d-b5d1ec8df1c1 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.303792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d56ca9a3-7398-4c38-837d-b5d1ec8df1c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.327242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:16.344507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56ca9a3-7398-4c38-837d-b5d1ec8df1c1 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.350126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.350964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.351344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.351889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] PATH_INFO: `/v3/users/0c6143cf4b2d469ba3c29369dc06abac` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.352368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3852fd59-bc11-4e67-a8a3-c9863537de77 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.353699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c6143cf4b2d469ba3c29369dc06abac)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.355861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.369786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8fab1-b3b3-4a06-aa81-1268fbbb7c4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.371062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8fab1-b3b3-4a06-aa81-1268fbbb7c4c admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.372627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8fab1-b3b3-4a06-aa81-1268fbbb7c4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.373072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8fab1-b3b3-4a06-aa81-1268fbbb7c4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.374012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc8fab1-b3b3-4a06-aa81-1268fbbb7c4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.378201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc8fab1-b3b3-4a06-aa81-1268fbbb7c4c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.440658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 937/1864] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:16 2022] POST /identity/v3/projects => generated 377 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:16.441826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8fab1-b3b3-4a06-aa81-1268fbbb7c4c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.464937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c6143cf4b2d469ba3c29369dc06abac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.480319 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c6143cf4b2d469ba3c29369dc06abac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.483310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c6143cf4b2d469ba3c29369dc06abac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.485329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c6143cf4b2d469ba3c29369dc06abac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.500740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c6143cf4b2d469ba3c29369dc06abac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:16.539820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 928/1865] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:16 2022] DELETE /identity/v3/users/0c6143cf4b2d469ba3c29369dc06abac => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:16.543326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.545218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.572071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8fab1-b3b3-4a06-aa81-1268fbbb7c4c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.705347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3195def6-8b3b-4c95-8bfe-845e8b09a167 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.711510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.712361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.715964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.716442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.717240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.719814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.730961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bc7a57-ce04-4385-8207-c0a90e48a0f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.731916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bc7a57-ce04-4385-8207-c0a90e48a0f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.732042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bc7a57-ce04-4385-8207-c0a90e48a0f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.732301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bc7a57-ce04-4385-8207-c0a90e48a0f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.732964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bc7a57-ce04-4385-8207-c0a90e48a0f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:16.733266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bc7a57-ce04-4385-8207-c0a90e48a0f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.736027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bc7a57-ce04-4385-8207-c0a90e48a0f2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.750318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 938/1866] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:16.751775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c4fbb0-fce5-489b-8240-00136d8f386a None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.752066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c4fbb0-fce5-489b-8240-00136d8f386a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.752507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c4fbb0-fce5-489b-8240-00136d8f386a None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.753615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 929/1867] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:16 2022] POST /identity/v3/users => generated 295 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:16.753989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 939/1868] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:10:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:10:16.754551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.756627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.757572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c4fbb0-fce5-489b-8240-00136d8f386a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.769756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62740b2e-1902-4f45-ab8c-9feb942a6cc0 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.774248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c4fbb0-fce5-489b-8240-00136d8f386a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.774486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3090319-57ac-448d-804a-968b9b15a007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.775130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3090319-57ac-448d-804a-968b9b15a007 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.775537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3090319-57ac-448d-804a-968b9b15a007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.775846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3090319-57ac-448d-804a-968b9b15a007 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.777017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3090319-57ac-448d-804a-968b9b15a007 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.778547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3090319-57ac-448d-804a-968b9b15a007 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.779877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3409208-a08e-41c0-81b3-510685188933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.781472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3409208-a08e-41c0-81b3-510685188933 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.781833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3409208-a08e-41c0-81b3-510685188933 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.782234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3409208-a08e-41c0-81b3-510685188933 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.782924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3409208-a08e-41c0-81b3-510685188933 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.785078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3409208-a08e-41c0-81b3-510685188933 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.803461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 940/1869] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:16 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:16.805457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402d0acc-c4b1-49fe-955c-fc6b4628451f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.805920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402d0acc-c4b1-49fe-955c-fc6b4628451f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.806199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402d0acc-c4b1-49fe-955c-fc6b4628451f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.826072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 930/1870] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:16 2022] POST /identity/v3/projects => generated 385 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:16.827123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3090319-57ac-448d-804a-968b9b15a007 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.829048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-402d0acc-c4b1-49fe-955c-fc6b4628451f None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:16.830636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-402d0acc-c4b1-49fe-955c-fc6b4628451f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.853504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:16.884696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=85,keystone:DELETE=10,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:16.913109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 941/1871] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:10:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:16.914028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402d0acc-c4b1-49fe-955c-fc6b4628451f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.925700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402d0acc-c4b1-49fe-955c-fc6b4628451f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.925916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3090319-57ac-448d-804a-968b9b15a007 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:16.931668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.932749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.932946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3974e1-f31c-46c9-bb12-78384b518d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:16.933071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.933245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.933795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3974e1-f31c-46c9-bb12-78384b518d36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:16.933913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.934084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3974e1-f31c-46c9-bb12-78384b518d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:16.934366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3974e1-f31c-46c9-bb12-78384b518d36 admin admin] PATH_INFO: `/v3/projects/27f196503e4248c9a58ce1b3eb62aecb/users/84455d7684614ed89f222ca3be662a2a/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:16.935850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.960139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3974e1-f31c-46c9-bb12-78384b518d36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27f196503e4248c9a58ce1b3eb62aecb, user_id=84455d7684614ed89f222ca3be662a2a, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:16.963180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3974e1-f31c-46c9-bb12-78384b518d36 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:16.967670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 942/1872] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:16 2022] POST /identity/v3/users => generated 293 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:16.968945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.971727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:16.978333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 931/1873] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:16 2022] PUT /identity/v3/projects/27f196503e4248c9a58ce1b3eb62aecb/users/84455d7684614ed89f222ca3be662a2a/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:16.979104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3974e1-f31c-46c9-bb12-78384b518d36 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:16.989377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6128f00f-c3ad-42cf-bec5-27928b95a8ba admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.074200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3974e1-f31c-46c9-bb12-78384b518d36 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.079411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ca907-a089-4a31-a1bc-ba7405b2d7cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.080186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ca907-a089-4a31-a1bc-ba7405b2d7cd admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.080571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ca907-a089-4a31-a1bc-ba7405b2d7cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.081775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ca907-a089-4a31-a1bc-ba7405b2d7cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.083348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51ca907-a089-4a31-a1bc-ba7405b2d7cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.085106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51ca907-a089-4a31-a1bc-ba7405b2d7cd admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.111917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 932/1874] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:16 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:17.111917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ca907-a089-4a31-a1bc-ba7405b2d7cd admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.112153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.115736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.115736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.115736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.122300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51ca907-a089-4a31-a1bc-ba7405b2d7cd admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.123511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.125313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.127443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407619cc-3a71-40f9-a33f-7a6380044957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.128156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407619cc-3a71-40f9-a33f-7a6380044957 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.128439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407619cc-3a71-40f9-a33f-7a6380044957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.128798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407619cc-3a71-40f9-a33f-7a6380044957 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.129438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407619cc-3a71-40f9-a33f-7a6380044957 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.131727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407619cc-3a71-40f9-a33f-7a6380044957 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.137581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 943/1875] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:17.138358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.140568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:17.157217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd889268-65dd-46ec-ace7-824b6ac4fefb service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.159291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 933/1876] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:17 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:17.160715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407619cc-3a71-40f9-a33f-7a6380044957 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.165658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.166294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.166591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.166856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] PATH_INFO: `/v3/projects/38b44c1bdab64428a8ccd8c43d9492f6` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.168029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] RBAC: Authorizing `identity:delete_project(project_id=38b44c1bdab64428a8ccd8c43d9492f6)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.170759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.174719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407619cc-3a71-40f9-a33f-7a6380044957 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.181945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636bec7d-593d-4ced-9a66-3cdbfb6e108f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.182721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636bec7d-593d-4ced-9a66-3cdbfb6e108f admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.183138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636bec7d-593d-4ced-9a66-3cdbfb6e108f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.183510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636bec7d-593d-4ced-9a66-3cdbfb6e108f admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.184344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636bec7d-593d-4ced-9a66-3cdbfb6e108f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.186137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636bec7d-593d-4ced-9a66-3cdbfb6e108f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.201610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 934/1877] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:17 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:17.203228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636bec7d-593d-4ced-9a66-3cdbfb6e108f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.213819 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636bec7d-593d-4ced-9a66-3cdbfb6e108f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.218611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccec9a-2b58-434f-a04d-3bd64379d96f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.219463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccec9a-2b58-434f-a04d-3bd64379d96f admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.219771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccec9a-2b58-434f-a04d-3bd64379d96f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.220045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccec9a-2b58-434f-a04d-3bd64379d96f admin admin] PATH_INFO: `/v3/projects/27f196503e4248c9a58ce1b3eb62aecb/users/84455d7684614ed89f222ca3be662a2a/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.221937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ccec9a-2b58-434f-a04d-3bd64379d96f admin admin] RBAC: Authorizing `identity:create_grant(project_id=27f196503e4248c9a58ce1b3eb62aecb, user_id=84455d7684614ed89f222ca3be662a2a, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.224885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ccec9a-2b58-434f-a04d-3bd64379d96f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.240373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 935/1878] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:17 2022] PUT /identity/v3/projects/27f196503e4248c9a58ce1b3eb62aecb/users/84455d7684614ed89f222ca3be662a2a/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:17.241179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccec9a-2b58-434f-a04d-3bd64379d96f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.250214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38b44c1bdab64428a8ccd8c43d9492f6', 'request_id': 'req-af8c5c85-1b01-4313-b346-091c04feadef'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:17.258016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38b44c1bdab64428a8ccd8c43d9492f6', 'request_id': 'req-af8c5c85-1b01-4313-b346-091c04feadef'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:17.263820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38b44c1bdab64428a8ccd8c43d9492f6', 'request_id': 'req-af8c5c85-1b01-4313-b346-091c04feadef'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:17.277065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] The token cache is being invalidate because project 38b44c1bdab64428a8ccd8c43d9492f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:17.277456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:17.278357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-af8c5c85-1b01-4313-b346-091c04feadef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:17.280120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 944/1879] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:17 2022] DELETE /identity/v3/projects/38b44c1bdab64428a8ccd8c43d9492f6 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:17.281334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20423880-be4e-4015-bebc-1f80828c278f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.282090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20423880-be4e-4015-bebc-1f80828c278f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.282372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20423880-be4e-4015-bebc-1f80828c278f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.290304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:17.319848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-20423880-be4e-4015-bebc-1f80828c278f None None] MFA Rules not processed for user `84455d7684614ed89f222ca3be662a2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:17.321107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-20423880-be4e-4015-bebc-1f80828c278f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:17.334382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccec9a-2b58-434f-a04d-3bd64379d96f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.384293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:17.421773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.422443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.422724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.422995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] PATH_INFO: `/v3/projects/7b10ec8d5067452a82192665db8b9e19/users/65060bef309f448298e9ee25ee5be87a/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.443765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b10ec8d5067452a82192665db8b9e19, user_id=65060bef309f448298e9ee25ee5be87a, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.447358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.475822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 945/1880] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:17 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:17.477461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05cbdc-8422-4b6c-a287-e757f9f25512 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.477747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05cbdc-8422-4b6c-a287-e757f9f25512 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.478030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05cbdc-8422-4b6c-a287-e757f9f25512 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.485047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 936/1881] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:17 2022] PUT /identity/v3/projects/7b10ec8d5067452a82192665db8b9e19/users/65060bef309f448298e9ee25ee5be87a/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:17.490627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.497200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:17.553314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9f05cbdc-8422-4b6c-a287-e757f9f25512 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:17.554436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9f05cbdc-8422-4b6c-a287-e757f9f25512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:17.649552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fdc452-2a7a-4425-b918-2e3c2bd5f8b0 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.652101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 946/1882] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:17.654516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42672b3e-c91c-4674-bb4d-839761985499 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.654876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42672b3e-c91c-4674-bb4d-839761985499 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.655817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42672b3e-c91c-4674-bb4d-839761985499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.673576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.674234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.675313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.675599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.690388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-42672b3e-c91c-4674-bb4d-839761985499 None None] MFA Rules not processed for user `65060bef309f448298e9ee25ee5be87a`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:17.693448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-42672b3e-c91c-4674-bb4d-839761985499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:17.782891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 947/1883] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:17 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:17.784081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568508c8-e045-473d-ad37-9fdc490159c1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.784401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568508c8-e045-473d-ad37-9fdc490159c1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.784648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568508c8-e045-473d-ad37-9fdc490159c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.811521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-568508c8-e045-473d-ad37-9fdc490159c1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:17.813702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-568508c8-e045-473d-ad37-9fdc490159c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:17.824155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 948/1884] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:17.829259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568508c8-e045-473d-ad37-9fdc490159c1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.842694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568508c8-e045-473d-ad37-9fdc490159c1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.845755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.847944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.864200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.864911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.865200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.865541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.866219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:17.866536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.870011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.873582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 937/1885] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:17 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:10:17.875649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.880904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:17.887795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:17.896957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 949/1886] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:17.900838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:17.912685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170d7597-3e61-42f6-a82d-506b1900efb7 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.920276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5563ac2-1d4f-4b49-9dc4-949113a1c563 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:17.927753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d077ba-de0d-4f8e-a72f-9eff5d368f40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.929288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d077ba-de0d-4f8e-a72f-9eff5d368f40 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.930365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d077ba-de0d-4f8e-a72f-9eff5d368f40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.930832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d077ba-de0d-4f8e-a72f-9eff5d368f40 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.930982 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb98472-b209-4e99-b1b3-ad6eceb79e2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:17.931618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d077ba-de0d-4f8e-a72f-9eff5d368f40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.931768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb98472-b209-4e99-b1b3-ad6eceb79e2e service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:17.932010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb98472-b209-4e99-b1b3-ad6eceb79e2e service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:17.932559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb98472-b209-4e99-b1b3-ad6eceb79e2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:17.934419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d077ba-de0d-4f8e-a72f-9eff5d368f40 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.954550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb98472-b209-4e99-b1b3-ad6eceb79e2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:17.956232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb98472-b209-4e99-b1b3-ad6eceb79e2e service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:17.983621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 938/1887] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:17 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:17.993964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 950/1888] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:17 2022] POST /identity/v3/projects => generated 401 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:17.998691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb98472-b209-4e99-b1b3-ad6eceb79e2e service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:18.014185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb98472-b209-4e99-b1b3-ad6eceb79e2e service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:18.083539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:18.096206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa46121-145f-41f7-8c78-483649c4d079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:18.097610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa46121-145f-41f7-8c78-483649c4d079 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:18.098378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa46121-145f-41f7-8c78-483649c4d079 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:18.098758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa46121-145f-41f7-8c78-483649c4d079 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:18.099763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa46121-145f-41f7-8c78-483649c4d079 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:18.101939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa46121-145f-41f7-8c78-483649c4d079 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:18.134981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 939/1889] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:17 2022] POST /identity/v3/users => generated 301 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:18.139087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d077ba-de0d-4f8e-a72f-9eff5d368f40 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:18.157005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d077ba-de0d-4f8e-a72f-9eff5d368f40 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:18.170965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29767293-0d23-43f0-907e-df9424357be0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:18.172612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29767293-0d23-43f0-907e-df9424357be0 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:18.172824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29767293-0d23-43f0-907e-df9424357be0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:18.173218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29767293-0d23-43f0-907e-df9424357be0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:18.173887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29767293-0d23-43f0-907e-df9424357be0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:18.180618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29767293-0d23-43f0-907e-df9424357be0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:18.204735 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 951/1890] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:18 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:18.209095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa46121-145f-41f7-8c78-483649c4d079 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:18.226188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa46121-145f-41f7-8c78-483649c4d079 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:18.240615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675916b2-a86b-496b-a96b-12ac4270fd7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:18.241274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675916b2-a86b-496b-a96b-12ac4270fd7c admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:18.241556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675916b2-a86b-496b-a96b-12ac4270fd7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:18.243104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675916b2-a86b-496b-a96b-12ac4270fd7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:18.244956 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675916b2-a86b-496b-a96b-12ac4270fd7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:18.246814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675916b2-a86b-496b-a96b-12ac4270fd7c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:18.272864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 940/1891] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:18 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:18.274938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29767293-0d23-43f0-907e-df9424357be0 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:18.292344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29767293-0d23-43f0-907e-df9424357be0 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:18.322798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34aad443-1f14-4837-a970-883942422979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:18.323484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aad443-1f14-4837-a970-883942422979 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:18.325377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aad443-1f14-4837-a970-883942422979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:18.325377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aad443-1f14-4837-a970-883942422979 admin admin] PATH_INFO: `/v3/projects/3f67022e1b904e5891c9a7af921f176f/users/8e65e5a26fec4d7692465e8383ae3162/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:18.362474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aad443-1f14-4837-a970-883942422979 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f67022e1b904e5891c9a7af921f176f, user_id=8e65e5a26fec4d7692465e8383ae3162, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:18.370043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aad443-1f14-4837-a970-883942422979 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:18.414568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675916b2-a86b-496b-a96b-12ac4270fd7c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:18.421091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 952/1892] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:18 2022] PUT /identity/v3/projects/3f67022e1b904e5891c9a7af921f176f/users/8e65e5a26fec4d7692465e8383ae3162/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:18.436398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a28547-5666-474a-84e2-fdc1c3804394 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:18.436796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a28547-5666-474a-84e2-fdc1c3804394 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:18.437808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a28547-5666-474a-84e2-fdc1c3804394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:18.545319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-56a28547-5666-474a-84e2-fdc1c3804394 None None] MFA Rules not processed for user `8e65e5a26fec4d7692465e8383ae3162`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:18.550895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-56a28547-5666-474a-84e2-fdc1c3804394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:18.581230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675916b2-a86b-496b-a96b-12ac4270fd7c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:18.684611 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 953/1893] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:18 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:18.685779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a28547-5666-474a-84e2-fdc1c3804394 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:18.758906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:18.759536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:18.759840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:18.761882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:18.765543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:18.873977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a28547-5666-474a-84e2-fdc1c3804394 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:18.884834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de2c5c4-63b0-48e4-80f2-a60d9532f2ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:18.886863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de2c5c4-63b0-48e4-80f2-a60d9532f2ad admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:18.887526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de2c5c4-63b0-48e4-80f2-a60d9532f2ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:18.889173 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de2c5c4-63b0-48e4-80f2-a60d9532f2ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:18.889737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de2c5c4-63b0-48e4-80f2-a60d9532f2ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:18.897100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de2c5c4-63b0-48e4-80f2-a60d9532f2ad admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:18.931431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:18.934786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:18.960808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 954/1894] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:18 2022] POST /identity/v3/projects => generated 399 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:18.961957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de2c5c4-63b0-48e4-80f2-a60d9532f2ad admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:18.987880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 941/1895] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:18 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 573 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:19.012698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:19.084140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de2c5c4-63b0-48e4-80f2-a60d9532f2ad admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:19.089943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d28b6-d770-43ea-afd3-dbb9846fd77f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:19.090200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d28b6-d770-43ea-afd3-dbb9846fd77f admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:19.090538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d28b6-d770-43ea-afd3-dbb9846fd77f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:19.090907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d28b6-d770-43ea-afd3-dbb9846fd77f admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:19.091673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9d28b6-d770-43ea-afd3-dbb9846fd77f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:19.093962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9d28b6-d770-43ea-afd3-dbb9846fd77f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:19.127405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 955/1896] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:18 2022] POST /identity/v3/users => generated 306 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:19.128431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:19.141101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58f99a9-cd81-4e47-ba3c-99be53335f9c service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:19.147663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7710a201-8f6c-43bd-9571-da3b80b21b28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:19.148318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7710a201-8f6c-43bd-9571-da3b80b21b28 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:19.148603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7710a201-8f6c-43bd-9571-da3b80b21b28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:19.148873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7710a201-8f6c-43bd-9571-da3b80b21b28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:19.149534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7710a201-8f6c-43bd-9571-da3b80b21b28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:19.152462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7710a201-8f6c-43bd-9571-da3b80b21b28 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:19.165074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 942/1897] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:10:19.169354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d28b6-d770-43ea-afd3-dbb9846fd77f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:19.184161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d28b6-d770-43ea-afd3-dbb9846fd77f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:19.191807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9127a24-54c0-40d6-982b-91f52e798e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:19.192666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9127a24-54c0-40d6-982b-91f52e798e8c admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:19.193194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9127a24-54c0-40d6-982b-91f52e798e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:19.193638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9127a24-54c0-40d6-982b-91f52e798e8c admin admin] PATH_INFO: `/v3/projects/b0d2ae984c084f2e8c6964177fde2678/users/c3d719ec7ce34d709a107ccbaf826e1e/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:19.215436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9127a24-54c0-40d6-982b-91f52e798e8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0d2ae984c084f2e8c6964177fde2678, user_id=c3d719ec7ce34d709a107ccbaf826e1e, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:19.219594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9127a24-54c0-40d6-982b-91f52e798e8c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:19.240733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 956/1898] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:19 2022] PUT /identity/v3/projects/b0d2ae984c084f2e8c6964177fde2678/users/c3d719ec7ce34d709a107ccbaf826e1e/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:19.242772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7710a201-8f6c-43bd-9571-da3b80b21b28 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:19.259632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7710a201-8f6c-43bd-9571-da3b80b21b28 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:19.338432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:19.401870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeda831-1c97-4ef7-ad71-bc791179e0e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:19.412473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeda831-1c97-4ef7-ad71-bc791179e0e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:19.412926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeda831-1c97-4ef7-ad71-bc791179e0e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:19.413214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeda831-1c97-4ef7-ad71-bc791179e0e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:19.414551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeda831-1c97-4ef7-ad71-bc791179e0e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:19.416992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeda831-1c97-4ef7-ad71-bc791179e0e2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:19.439916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 943/1899] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:19.441285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9127a24-54c0-40d6-982b-91f52e798e8c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:19.454314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9127a24-54c0-40d6-982b-91f52e798e8c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:19.461761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6199fea4-a427-452d-b1d0-5402897d4616 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:19.462933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6199fea4-a427-452d-b1d0-5402897d4616 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:19.463217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6199fea4-a427-452d-b1d0-5402897d4616 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:19.463486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6199fea4-a427-452d-b1d0-5402897d4616 admin admin] PATH_INFO: `/v3/projects/b0d2ae984c084f2e8c6964177fde2678/users/c3d719ec7ce34d709a107ccbaf826e1e/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:19.465818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6199fea4-a427-452d-b1d0-5402897d4616 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0d2ae984c084f2e8c6964177fde2678, user_id=c3d719ec7ce34d709a107ccbaf826e1e, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:19.468682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6199fea4-a427-452d-b1d0-5402897d4616 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:19.480853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 957/1900] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:19 2022] PUT /identity/v3/projects/b0d2ae984c084f2e8c6964177fde2678/users/c3d719ec7ce34d709a107ccbaf826e1e/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:19.491657 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa68cb-ba83-4d72-aa7c-bac956903a2c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:19.491964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa68cb-ba83-4d72-aa7c-bac956903a2c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:19.492243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa68cb-ba83-4d72-aa7c-bac956903a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:19.549307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d4aa68cb-ba83-4d72-aa7c-bac956903a2c None None] MFA Rules not processed for user `c3d719ec7ce34d709a107ccbaf826e1e`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:19.553772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d4aa68cb-ba83-4d72-aa7c-bac956903a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:19.661365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 944/1901] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:19 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:19.672156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6199fea4-a427-452d-b1d0-5402897d4616 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:19.747173 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6199fea4-a427-452d-b1d0-5402897d4616 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:19.754917 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:19.756209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:19.756729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:19.757458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:19.786320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:19.787797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:19.799234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 958/1902] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:19 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:19.927595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa68cb-ba83-4d72-aa7c-bac956903a2c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:19.946887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa68cb-ba83-4d72-aa7c-bac956903a2c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:19.957375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7260f5-2b34-4b0b-814f-324beb92696d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:19.958076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7260f5-2b34-4b0b-814f-324beb92696d service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:19.958427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7260f5-2b34-4b0b-814f-324beb92696d service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:19.958764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7260f5-2b34-4b0b-814f-324beb92696d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:20.118706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7260f5-2b34-4b0b-814f-324beb92696d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:20.120613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7260f5-2b34-4b0b-814f-324beb92696d service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:20.140698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 945/1903] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:19 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:26.708583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:26.711073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:26.847509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab8b82-e9fd-4c7f-843d-64d699d1d787 service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:26.852790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:26.853634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:26.854209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:26.854772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] PATH_INFO: `/v3/users/0d6be58caa1246f7b3819c5ec8d3b8b6` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:26.856145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d6be58caa1246f7b3819c5ec8d3b8b6)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:26.858246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:26.882000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:26.968382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d6be58caa1246f7b3819c5ec8d3b8b6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:26.982843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d6be58caa1246f7b3819c5ec8d3b8b6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:26.984104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d6be58caa1246f7b3819c5ec8d3b8b6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:26.985585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d6be58caa1246f7b3819c5ec8d3b8b6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:26.996421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d6be58caa1246f7b3819c5ec8d3b8b6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:27.030549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 959/1904] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:26 2022] DELETE /identity/v3/users/0d6be58caa1246f7b3819c5ec8d3b8b6 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:27.035350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057d80f2-3753-479b-9148-323b93353672 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:27.035854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057d80f2-3753-479b-9148-323b93353672 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:27.036075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057d80f2-3753-479b-9148-323b93353672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:27.062871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-057d80f2-3753-479b-9148-323b93353672 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:27.064252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-057d80f2-3753-479b-9148-323b93353672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:27.114892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:27.154060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 946/1905] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:27.164490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:27.165907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:27.261209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87add78-a521-4853-a250-66a0a13678c5 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:27.266708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6e934-d9e9-495d-887f-e205e5615a44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:27.267275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6e934-d9e9-495d-887f-e205e5615a44 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:27.267672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6e934-d9e9-495d-887f-e205e5615a44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:27.267858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6e934-d9e9-495d-887f-e205e5615a44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:27.278157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e6e934-d9e9-495d-887f-e205e5615a44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:27.279787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e6e934-d9e9-495d-887f-e205e5615a44 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:27.291346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 960/1906] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:27.765518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057d80f2-3753-479b-9148-323b93353672 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:27.767404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-057d80f2-3753-479b-9148-323b93353672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:27.786839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057d80f2-3753-479b-9148-323b93353672 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:27.828249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:27.829106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:27.829463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:27.829875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] PATH_INFO: `/v3/projects/9d8a20069ce746b4a0e105f515a812f5` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:27.832375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d8a20069ce746b4a0e105f515a812f5)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:27.835472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:27.915102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d8a20069ce746b4a0e105f515a812f5', 'request_id': 'req-faf00c33-7503-4c7f-95cb-6b578118c226'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:27.930252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d8a20069ce746b4a0e105f515a812f5', 'request_id': 'req-faf00c33-7503-4c7f-95cb-6b578118c226'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:27.936857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d8a20069ce746b4a0e105f515a812f5', 'request_id': 'req-faf00c33-7503-4c7f-95cb-6b578118c226'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:27.951035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] The token cache is being invalidate because project 9d8a20069ce746b4a0e105f515a812f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:27.951330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:27.952016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:27.957048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 947/1907] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:27 2022] DELETE /identity/v3/projects/9d8a20069ce746b4a0e105f515a812f5 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:27.958312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6e934-d9e9-495d-887f-e205e5615a44 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:27.969955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6e934-d9e9-495d-887f-e205e5615a44 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:28.034016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:28.034655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:28.034931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:28.035206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] PATH_INFO: `/v3/users/742d680f400e4f69a7768aad9da289c5` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:28.036547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=742d680f400e4f69a7768aad9da289c5)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:28.038486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:28.085648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:28.096052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '742d680f400e4f69a7768aad9da289c5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.103450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '742d680f400e4f69a7768aad9da289c5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.104076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '742d680f400e4f69a7768aad9da289c5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.104681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '742d680f400e4f69a7768aad9da289c5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.112480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '742d680f400e4f69a7768aad9da289c5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.129936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 961/1908] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:27 2022] DELETE /identity/v3/users/742d680f400e4f69a7768aad9da289c5 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:28.509904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:28.511370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:28.525424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf00c33-7503-4c7f-95cb-6b578118c226 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:28.583058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:28.583711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:28.584010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:28.584289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] PATH_INFO: `/v3/projects/f7e31117dc6042c19036459c82f15316` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:28.585413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7e31117dc6042c19036459c82f15316)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:28.587331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:28.632446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7e31117dc6042c19036459c82f15316', 'request_id': 'req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.639557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7e31117dc6042c19036459c82f15316', 'request_id': 'req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.645827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7e31117dc6042c19036459c82f15316', 'request_id': 'req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.657343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] The token cache is being invalidate because project f7e31117dc6042c19036459c82f15316 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:28.657635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.658272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.659771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 948/1909] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:28 2022] DELETE /identity/v3/projects/f7e31117dc6042c19036459c82f15316 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:28.663711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:28.665135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:28.677978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98368d8a-e091-4072-b60e-96a93179dd1a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:28.736534 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:28.737135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:28.737427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:28.737684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] PATH_INFO: `/v3/users/84455d7684614ed89f222ca3be662a2a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:28.738924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] RBAC: Authorizing `identity:delete_user(user_id=84455d7684614ed89f222ca3be662a2a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:28.746214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:28.795084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84455d7684614ed89f222ca3be662a2a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.802778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84455d7684614ed89f222ca3be662a2a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.803420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84455d7684614ed89f222ca3be662a2a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.804030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84455d7684614ed89f222ca3be662a2a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.811250 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2cfe68f2-f4e9-46c3-92d2-cab3a20aa704 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84455d7684614ed89f222ca3be662a2a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:28.829239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 962/1910] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:28 2022] DELETE /identity/v3/users/84455d7684614ed89f222ca3be662a2a => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:29.173718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:29.175429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:29.197431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ce8e08-72ac-4b74-aaa0-d99ab4938f80 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:29.201978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:29.266396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:29.267138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.267420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.267707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] PATH_INFO: `/v3/projects/27f196503e4248c9a58ce1b3eb62aecb` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.268833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] RBAC: Authorizing `identity:delete_project(project_id=27f196503e4248c9a58ce1b3eb62aecb)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:29.270733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:29.318323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27f196503e4248c9a58ce1b3eb62aecb', 'request_id': 'req-03363a82-cf22-486e-af77-884134ded22b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:29.325584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27f196503e4248c9a58ce1b3eb62aecb', 'request_id': 'req-03363a82-cf22-486e-af77-884134ded22b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:29.331531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27f196503e4248c9a58ce1b3eb62aecb', 'request_id': 'req-03363a82-cf22-486e-af77-884134ded22b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:29.342496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] The token cache is being invalidate because project 27f196503e4248c9a58ce1b3eb62aecb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:29.342782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:29.344108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-03363a82-cf22-486e-af77-884134ded22b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:29.345740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 949/1911] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:29 2022] DELETE /identity/v3/projects/27f196503e4248c9a58ce1b3eb62aecb => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:29.351898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ee0f31-e934-43f6-9937-b3fd5ef4902a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.352364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ee0f31-e934-43f6-9937-b3fd5ef4902a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.352470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ee0f31-e934-43f6-9937-b3fd5ef4902a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.372945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c3ee0f31-e934-43f6-9937-b3fd5ef4902a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:29.374146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c3ee0f31-e934-43f6-9937-b3fd5ef4902a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:29.403400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:29.435908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 963/1912] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:29.446058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff6221a-fe74-4c73-a4c2-0ff18d758b92 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.446344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff6221a-fe74-4c73-a4c2-0ff18d758b92 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.446612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff6221a-fe74-4c73-a4c2-0ff18d758b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.466969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-7ff6221a-fe74-4c73-a4c2-0ff18d758b92 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:29.468284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7ff6221a-fe74-4c73-a4c2-0ff18d758b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:29.476779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 950/1913] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:29.480902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ee0f31-e934-43f6-9937-b3fd5ef4902a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:29.491589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ee0f31-e934-43f6-9937-b3fd5ef4902a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:29.496361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:29.496971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.497256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.497535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.498184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:29.498486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:29.500713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:29.513040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 964/1914] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:29.517461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff6221a-fe74-4c73-a4c2-0ff18d758b92 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:29.527480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff6221a-fe74-4c73-a4c2-0ff18d758b92 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:29.533997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7a19ba-e336-41f4-9ae0-7938a423b193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:29.534863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7a19ba-e336-41f4-9ae0-7938a423b193 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.535320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7a19ba-e336-41f4-9ae0-7938a423b193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.535822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7a19ba-e336-41f4-9ae0-7938a423b193 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.536648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7a19ba-e336-41f4-9ae0-7938a423b193 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:29.539072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7a19ba-e336-41f4-9ae0-7938a423b193 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:29.574913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 951/1915] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:29 2022] POST /identity/v3/projects => generated 365 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:29.580408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:29.590689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361360c8-ce73-40bd-b2da-84c984d71f14 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:29.650491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58a7224-7f35-415d-bd0e-93dbc66f5cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:29.651101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58a7224-7f35-415d-bd0e-93dbc66f5cd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.651383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58a7224-7f35-415d-bd0e-93dbc66f5cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.651660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58a7224-7f35-415d-bd0e-93dbc66f5cd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.652312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58a7224-7f35-415d-bd0e-93dbc66f5cd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:29.654230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58a7224-7f35-415d-bd0e-93dbc66f5cd7 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:29.675153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 965/1916] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:29 2022] POST /identity/v3/users => generated 283 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:10:29.679331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7a19ba-e336-41f4-9ae0-7938a423b193 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:29.689851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7a19ba-e336-41f4-9ae0-7938a423b193 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:29.694387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a8a2fe-6dda-4231-b46a-5b17b45db4b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:29.694993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a8a2fe-6dda-4231-b46a-5b17b45db4b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.695273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a8a2fe-6dda-4231-b46a-5b17b45db4b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.695547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a8a2fe-6dda-4231-b46a-5b17b45db4b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.696230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a8a2fe-6dda-4231-b46a-5b17b45db4b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:29.697817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a8a2fe-6dda-4231-b46a-5b17b45db4b9 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:29.708315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 952/1917] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:29.712112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58a7224-7f35-415d-bd0e-93dbc66f5cd7 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:29.722539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58a7224-7f35-415d-bd0e-93dbc66f5cd7 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:29.727291 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5ce17e-4f53-4ed9-a88e-f4f0efaea2dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:29.727931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ce17e-4f53-4ed9-a88e-f4f0efaea2dd admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.728214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ce17e-4f53-4ed9-a88e-f4f0efaea2dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.728487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ce17e-4f53-4ed9-a88e-f4f0efaea2dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.729135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5ce17e-4f53-4ed9-a88e-f4f0efaea2dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:29.730758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5ce17e-4f53-4ed9-a88e-f4f0efaea2dd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:29.741371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 966/1918] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:29.745421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a8a2fe-6dda-4231-b46a-5b17b45db4b9 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:29.755062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a8a2fe-6dda-4231-b46a-5b17b45db4b9 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:29.759877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6240256-e051-48f6-8369-33539e41c8cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:29.760526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6240256-e051-48f6-8369-33539e41c8cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.760889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6240256-e051-48f6-8369-33539e41c8cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.761072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6240256-e051-48f6-8369-33539e41c8cf admin admin] PATH_INFO: `/v3/projects/7190d2ccbbe8414cadd765ae26c2946b/users/995acbc8292a4335ab13e4928cb37aa8/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.776477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6240256-e051-48f6-8369-33539e41c8cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=7190d2ccbbe8414cadd765ae26c2946b, user_id=995acbc8292a4335ab13e4928cb37aa8, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:29.779136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6240256-e051-48f6-8369-33539e41c8cf admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:29.793322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 953/1919] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:29 2022] PUT /identity/v3/projects/7190d2ccbbe8414cadd765ae26c2946b/users/995acbc8292a4335ab13e4928cb37aa8/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:29.804662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d004a0-f046-438e-a238-6727b6a60d19 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.804952 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d004a0-f046-438e-a238-6727b6a60d19 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.805230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d004a0-f046-438e-a238-6727b6a60d19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.833493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-01d004a0-f046-438e-a238-6727b6a60d19 None None] MFA Rules not processed for user `995acbc8292a4335ab13e4928cb37aa8`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:29.834694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-01d004a0-f046-438e-a238-6727b6a60d19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:29.897673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 967/1920] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:29 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:29.902350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6240256-e051-48f6-8369-33539e41c8cf admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:29.936445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:29.967786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6240256-e051-48f6-8369-33539e41c8cf admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:29.972557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf266ef-c641-49f5-aaed-6b94d72c6bff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:29.973188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf266ef-c641-49f5-aaed-6b94d72c6bff admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:29.973469 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf266ef-c641-49f5-aaed-6b94d72c6bff admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:29.973743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf266ef-c641-49f5-aaed-6b94d72c6bff admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:29.974382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf266ef-c641-49f5-aaed-6b94d72c6bff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:29.976391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf266ef-c641-49f5-aaed-6b94d72c6bff admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:30.006940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 954/1921] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:29 2022] POST /identity/v3/projects => generated 367 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:30.007997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d004a0-f046-438e-a238-6727b6a60d19 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:30.020037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d004a0-f046-438e-a238-6727b6a60d19 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:30.049842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:30.082098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a0e398-1bed-4fe4-ba6d-df44686cd8af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:30.082705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0e398-1bed-4fe4-ba6d-df44686cd8af admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:30.082986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0e398-1bed-4fe4-ba6d-df44686cd8af admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:30.083257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0e398-1bed-4fe4-ba6d-df44686cd8af admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:30.083933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a0e398-1bed-4fe4-ba6d-df44686cd8af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:30.085849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a0e398-1bed-4fe4-ba6d-df44686cd8af admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:30.106282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 968/1922] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:30 2022] POST /identity/v3/users => generated 290 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:10:30.110497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf266ef-c641-49f5-aaed-6b94d72c6bff admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:30.121281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf266ef-c641-49f5-aaed-6b94d72c6bff admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:30.126751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a3c9e8-3bef-43c9-85f5-f5495ad9172b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:30.127448 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a3c9e8-3bef-43c9-85f5-f5495ad9172b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:30.127839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a3c9e8-3bef-43c9-85f5-f5495ad9172b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:30.128119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a3c9e8-3bef-43c9-85f5-f5495ad9172b admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:30.128763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a3c9e8-3bef-43c9-85f5-f5495ad9172b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:30.130372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a3c9e8-3bef-43c9-85f5-f5495ad9172b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:30.142121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 955/1923] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:30.146181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a0e398-1bed-4fe4-ba6d-df44686cd8af admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:30.156073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a0e398-1bed-4fe4-ba6d-df44686cd8af admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:30.160722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15824962-709b-4330-b010-388cb2a2f280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:30.161310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15824962-709b-4330-b010-388cb2a2f280 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:30.161588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15824962-709b-4330-b010-388cb2a2f280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:30.161861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15824962-709b-4330-b010-388cb2a2f280 admin admin] PATH_INFO: `/v3/projects/bba77e8815c44481868cff88657abcd8/users/3f75554affb94f0b9f12438ab162aa96/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:30.176753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15824962-709b-4330-b010-388cb2a2f280 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bba77e8815c44481868cff88657abcd8, user_id=3f75554affb94f0b9f12438ab162aa96, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:30.179415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15824962-709b-4330-b010-388cb2a2f280 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:30.192928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 969/1924] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:30 2022] PUT /identity/v3/projects/bba77e8815c44481868cff88657abcd8/users/3f75554affb94f0b9f12438ab162aa96/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:30.196826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a3c9e8-3bef-43c9-85f5-f5495ad9172b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:30.207351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a3c9e8-3bef-43c9-85f5-f5495ad9172b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:30.266093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974c3a2b-64c1-45c2-9f1f-2a4cb7937d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:30.266765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c3a2b-64c1-45c2-9f1f-2a4cb7937d19 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:30.267016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c3a2b-64c1-45c2-9f1f-2a4cb7937d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:30.268086 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c3a2b-64c1-45c2-9f1f-2a4cb7937d19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:30.268755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974c3a2b-64c1-45c2-9f1f-2a4cb7937d19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:30.270459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974c3a2b-64c1-45c2-9f1f-2a4cb7937d19 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:30.281439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 956/1925] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:30 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:30.287648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15824962-709b-4330-b010-388cb2a2f280 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:30.303631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15824962-709b-4330-b010-388cb2a2f280 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:30.308685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:30.309343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:30.309620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:30.315800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] PATH_INFO: `/v3/projects/bba77e8815c44481868cff88657abcd8/users/3f75554affb94f0b9f12438ab162aa96/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:30.321234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bba77e8815c44481868cff88657abcd8, user_id=3f75554affb94f0b9f12438ab162aa96, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:30.326077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:30.339375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 970/1926] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:30 2022] PUT /identity/v3/projects/bba77e8815c44481868cff88657abcd8/users/3f75554affb94f0b9f12438ab162aa96/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:30.350573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd84c3-abeb-4d06-ac66-e38b8e87e9be None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:30.350889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd84c3-abeb-4d06-ac66-e38b8e87e9be None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:30.351157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd84c3-abeb-4d06-ac66-e38b8e87e9be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:30.383202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d7fd84c3-abeb-4d06-ac66-e38b8e87e9be None None] MFA Rules not processed for user `3f75554affb94f0b9f12438ab162aa96`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:30.384478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d7fd84c3-abeb-4d06-ac66-e38b8e87e9be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:30.462484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 957/1927] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:30 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:30.473866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:30.475606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:30.532912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257b4a8d-354d-43e7-ba0e-93e11f42bf66 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:30.536402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999846f1-cdd4-40e6-b764-713d88f51807 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:30.536995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999846f1-cdd4-40e6-b764-713d88f51807 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:30.537276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999846f1-cdd4-40e6-b764-713d88f51807 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:30.537554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999846f1-cdd4-40e6-b764-713d88f51807 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:30.546861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999846f1-cdd4-40e6-b764-713d88f51807 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:30.548365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999846f1-cdd4-40e6-b764-713d88f51807 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:30.557825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 971/1928] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:30 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:30.644798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fd84c3-abeb-4d06-ac66-e38b8e87e9be None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:30.654101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fd84c3-abeb-4d06-ac66-e38b8e87e9be None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:30.659368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75c5a36-6180-4b73-a932-dbd031c7e535 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:30.660184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75c5a36-6180-4b73-a932-dbd031c7e535 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:30.660470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75c5a36-6180-4b73-a932-dbd031c7e535 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:30.660745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75c5a36-6180-4b73-a932-dbd031c7e535 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:30.751268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:30.805452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75c5a36-6180-4b73-a932-dbd031c7e535 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:30.806799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75c5a36-6180-4b73-a932-dbd031c7e535 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:30.821286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 958/1929] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:30 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:37.358523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda63b40-1948-4c08-86df-42f08e6a46ea None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:37.359294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda63b40-1948-4c08-86df-42f08e6a46ea None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:37.359294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda63b40-1948-4c08-86df-42f08e6a46ea None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:37.361135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 972/1930] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:10:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:10:37.372035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf13ff83-082b-48b0-b70c-00a92f1b63c8 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:37.372528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf13ff83-082b-48b0-b70c-00a92f1b63c8 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:37.372673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf13ff83-082b-48b0-b70c-00a92f1b63c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:37.416126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-cf13ff83-082b-48b0-b70c-00a92f1b63c8 None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:37.417344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-cf13ff83-082b-48b0-b70c-00a92f1b63c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:37.546848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 959/1931] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:10:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:37.563607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda63b40-1948-4c08-86df-42f08e6a46ea None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:37.566062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bda63b40-1948-4c08-86df-42f08e6a46ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:37.587901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda63b40-1948-4c08-86df-42f08e6a46ea None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:37.595187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:37.595846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:37.596127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:37.596400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:37.608145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:37.611090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:37.622975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 973/1932] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:38.616383 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf13ff83-082b-48b0-b70c-00a92f1b63c8 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:38.619182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-cf13ff83-082b-48b0-b70c-00a92f1b63c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:38.653139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:38.655552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:38.720060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf13ff83-082b-48b0-b70c-00a92f1b63c8 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:38.728636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:38.729294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:38.729580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:38.729858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] PATH_INFO: `/v3/users/8e65e5a26fec4d7692465e8383ae3162` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:38.731176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e65e5a26fec4d7692465e8383ae3162)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:38.733641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:38.798466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e794ad-0ed1-472f-a5ca-20bd37546a34 service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:38.803705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:38.804409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:38.804704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:38.804988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] PATH_INFO: `/v3/users/65060bef309f448298e9ee25ee5be87a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:38.806130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65060bef309f448298e9ee25ee5be87a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:38.808748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:38.815765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e65e5a26fec4d7692465e8383ae3162'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.825204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:38.827117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e65e5a26fec4d7692465e8383ae3162'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.829138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e65e5a26fec4d7692465e8383ae3162'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.832331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e65e5a26fec4d7692465e8383ae3162'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.846050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:38.848923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-85fb80c6-2905-4d31-9e91-ef2ea7e0db65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e65e5a26fec4d7692465e8383ae3162'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.886855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65060bef309f448298e9ee25ee5be87a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.891989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 960/1933] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:38 2022] DELETE /identity/v3/users/8e65e5a26fec4d7692465e8383ae3162 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:38.894836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b935b8-20f0-489a-955e-558bfcf29ecd None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:38.895640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b935b8-20f0-489a-955e-558bfcf29ecd None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:38.896366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b935b8-20f0-489a-955e-558bfcf29ecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:38.905300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65060bef309f448298e9ee25ee5be87a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.906053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65060bef309f448298e9ee25ee5be87a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.906725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65060bef309f448298e9ee25ee5be87a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.918493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41f834f8-65d4-485a-a4a1-68c0519190c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65060bef309f448298e9ee25ee5be87a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:38.919305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c4b935b8-20f0-489a-955e-558bfcf29ecd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:38.924549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c4b935b8-20f0-489a-955e-558bfcf29ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:38.953993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 974/1934] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:38 2022] DELETE /identity/v3/users/65060bef309f448298e9ee25ee5be87a => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:38.958484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8686e4-4938-4c68-8732-53bdd95f83df None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:38.960083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8686e4-4938-4c68-8732-53bdd95f83df None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:38.960430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8686e4-4938-4c68-8732-53bdd95f83df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:38.990618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3d8686e4-4938-4c68-8732-53bdd95f83df None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:38.992360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3d8686e4-4938-4c68-8732-53bdd95f83df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:39.038570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 961/1935] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:39.050707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b935b8-20f0-489a-955e-558bfcf29ecd None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:39.052230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c4b935b8-20f0-489a-955e-558bfcf29ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:39.115122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 975/1936] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:39.116344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8686e4-4938-4c68-8732-53bdd95f83df None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:39.118319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3d8686e4-4938-4c68-8732-53bdd95f83df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:39.137644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b935b8-20f0-489a-955e-558bfcf29ecd None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:39.201722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8686e4-4938-4c68-8732-53bdd95f83df None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:39.207285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:39.208034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:39.208233 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:39.208526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:39.213023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:39.213967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:39.214429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:39.215034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:39.219627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:39.221363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:39.233500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:39.235675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:39.239138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 976/1937] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:39.250923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 962/1938] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:10:39.600044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:39.602256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:39.628213 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ae1972-5ff7-4492-a99e-bb43389f7301 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:39.636129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:39.636825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:39.637051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:39.637329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] PATH_INFO: `/v3/projects/3f67022e1b904e5891c9a7af921f176f` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:39.638485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f67022e1b904e5891c9a7af921f176f)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:39.641645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:39.671670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:39.673665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:39.684248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:39.694285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ae1bb9-3b8b-49e6-b647-fbec3b353fbd service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:39.704695 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f67022e1b904e5891c9a7af921f176f', 'request_id': 'req-e0a88262-c438-422c-9e4a-12ec31de92d5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.715575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f67022e1b904e5891c9a7af921f176f', 'request_id': 'req-e0a88262-c438-422c-9e4a-12ec31de92d5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.722531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f67022e1b904e5891c9a7af921f176f', 'request_id': 'req-e0a88262-c438-422c-9e4a-12ec31de92d5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.736486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] The token cache is being invalidate because project 3f67022e1b904e5891c9a7af921f176f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:39.736808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.737623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.740196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 977/1939] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:39 2022] DELETE /identity/v3/projects/3f67022e1b904e5891c9a7af921f176f => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:39.743814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:39.744943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:39.746741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:39.786878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:39.788638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:39.789213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:39.790089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] PATH_INFO: `/v3/projects/7b10ec8d5067452a82192665db8b9e19` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:39.791870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b10ec8d5067452a82192665db8b9e19)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:39.794969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:39.852455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a88262-c438-422c-9e4a-12ec31de92d5 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:39.870395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:39.871265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:39.871748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:39.872190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] PATH_INFO: `/v3/users/c3d719ec7ce34d709a107ccbaf826e1e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:39.873711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3d719ec7ce34d709a107ccbaf826e1e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:39.879639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:39.884948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b10ec8d5067452a82192665db8b9e19', 'request_id': 'req-4591df2f-5b1c-4906-8571-932f48a09c71'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.896172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b10ec8d5067452a82192665db8b9e19', 'request_id': 'req-4591df2f-5b1c-4906-8571-932f48a09c71'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.906371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b10ec8d5067452a82192665db8b9e19', 'request_id': 'req-4591df2f-5b1c-4906-8571-932f48a09c71'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.924221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] The token cache is being invalidate because project 7b10ec8d5067452a82192665db8b9e19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:39.924653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.925401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4591df2f-5b1c-4906-8571-932f48a09c71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:39.927078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 963/1940] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:39 2022] DELETE /identity/v3/projects/7b10ec8d5067452a82192665db8b9e19 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:39.933588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6575f602-c181-4607-94fd-663146909f7a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:39.933961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6575f602-c181-4607-94fd-663146909f7a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:39.935365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6575f602-c181-4607-94fd-663146909f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:39.965430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6575f602-c181-4607-94fd-663146909f7a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:39.967202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6575f602-c181-4607-94fd-663146909f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:40.048236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3d719ec7ce34d709a107ccbaf826e1e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:40.072853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3d719ec7ce34d709a107ccbaf826e1e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:40.072853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3d719ec7ce34d709a107ccbaf826e1e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:40.072853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3d719ec7ce34d709a107ccbaf826e1e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:40.104696 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3d719ec7ce34d709a107ccbaf826e1e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:40.113606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 964/1941] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:10:40.122882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7e263-3832-4c7d-a30f-70231e61eece None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:40.123175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7e263-3832-4c7d-a30f-70231e61eece None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:40.123453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7e263-3832-4c7d-a30f-70231e61eece None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:40.171944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 978/1942] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:39 2022] DELETE /identity/v3/users/c3d719ec7ce34d709a107ccbaf826e1e => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:40.193463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c0f7e263-3832-4c7d-a30f-70231e61eece None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:40.201642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c0f7e263-3832-4c7d-a30f-70231e61eece None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:40.323722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 965/1943] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:40.329839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:40.349187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01bea13-006a-4bb8-8828-047ba4167b5b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:40.356133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:40.356769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:40.357206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:40.357502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:40.358193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:40.358604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:40.360817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:40.381287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 979/1944] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:40.385945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f7e263-3832-4c7d-a30f-70231e61eece None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:40.397449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f7e263-3832-4c7d-a30f-70231e61eece None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:40.420185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daf5f78-c981-4421-9452-582e810ec1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:40.421169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daf5f78-c981-4421-9452-582e810ec1fa admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:40.424040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daf5f78-c981-4421-9452-582e810ec1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:40.425489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daf5f78-c981-4421-9452-582e810ec1fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:40.426335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daf5f78-c981-4421-9452-582e810ec1fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:40.430849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daf5f78-c981-4421-9452-582e810ec1fa admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:40.460441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:40.522195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 966/1945] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:40 2022] POST /identity/v3/projects => generated 371 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:40.526353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:40.540028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8a2876-e2ec-491e-84b8-962833428436 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:40.564852 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:40.618751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a10b866-80cc-4a9b-a7fa-e31bb34e50fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:40.620018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a10b866-80cc-4a9b-a7fa-e31bb34e50fd admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:40.620307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a10b866-80cc-4a9b-a7fa-e31bb34e50fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:40.620863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a10b866-80cc-4a9b-a7fa-e31bb34e50fd admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:40.621405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a10b866-80cc-4a9b-a7fa-e31bb34e50fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:40.623646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a10b866-80cc-4a9b-a7fa-e31bb34e50fd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:40.647861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 980/1946] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:40 2022] POST /identity/v3/users => generated 286 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:40.651060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daf5f78-c981-4421-9452-582e810ec1fa admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:40.661317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daf5f78-c981-4421-9452-582e810ec1fa admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:40.665985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b970497a-4aea-4084-a630-c3c495c8857a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:40.666592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970497a-4aea-4084-a630-c3c495c8857a admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:40.666869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970497a-4aea-4084-a630-c3c495c8857a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:40.667144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b970497a-4aea-4084-a630-c3c495c8857a admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:40.668370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b970497a-4aea-4084-a630-c3c495c8857a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:40.669610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b970497a-4aea-4084-a630-c3c495c8857a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:40.680541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 967/1947] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:40.684683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a10b866-80cc-4a9b-a7fa-e31bb34e50fd admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:40.698458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a10b866-80cc-4a9b-a7fa-e31bb34e50fd admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:40.704825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d9b305-b643-4b14-843e-494d96092d66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:40.706239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9b305-b643-4b14-843e-494d96092d66 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:40.706523 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9b305-b643-4b14-843e-494d96092d66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:40.706800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9b305-b643-4b14-843e-494d96092d66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:40.707456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d9b305-b643-4b14-843e-494d96092d66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:40.709347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d9b305-b643-4b14-843e-494d96092d66 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:40.722686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 981/1948] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:40.726783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b970497a-4aea-4084-a630-c3c495c8857a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:40.738736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b970497a-4aea-4084-a630-c3c495c8857a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:40.745804 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:40.746862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:40.747145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:40.747946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] PATH_INFO: `/v3/projects/95fe1ad03f21497aad64598e8f6d13f7/users/868dee55d493496090988ff601c7d7c3/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:40.770066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] RBAC: Authorizing `identity:create_grant(project_id=95fe1ad03f21497aad64598e8f6d13f7, user_id=868dee55d493496090988ff601c7d7c3, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:40.773603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:40.793085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 968/1949] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:40 2022] PUT /identity/v3/projects/95fe1ad03f21497aad64598e8f6d13f7/users/868dee55d493496090988ff601c7d7c3/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:40.804557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc0762-42dd-4ba0-8ea2-06f812648a5e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:40.804951 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc0762-42dd-4ba0-8ea2-06f812648a5e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:40.805337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc0762-42dd-4ba0-8ea2-06f812648a5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:40.866913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-59fc0762-42dd-4ba0-8ea2-06f812648a5e None None] MFA Rules not processed for user `868dee55d493496090988ff601c7d7c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:40.868117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-59fc0762-42dd-4ba0-8ea2-06f812648a5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:40.962359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 982/1950] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:40.970806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:40.972845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:41.093783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4583261-7b4f-4062-a055-bab0b082115a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:41.099725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d2b85c-db8b-461f-a79b-096662a6c57a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '54db2e6805fd40f79c03eb3b2319f2b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:41.099914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d2b85c-db8b-461f-a79b-096662a6c57a service glance] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:41.100130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d2b85c-db8b-461f-a79b-096662a6c57a service glance] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:41.100328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d2b85c-db8b-461f-a79b-096662a6c57a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:41.119885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d2b85c-db8b-461f-a79b-096662a6c57a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:41.122382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d2b85c-db8b-461f-a79b-096662a6c57a service glance] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:41.134438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 969/1951] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:40 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:41.645245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc0762-42dd-4ba0-8ea2-06f812648a5e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:41.646986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-59fc0762-42dd-4ba0-8ea2-06f812648a5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:41.808782 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc0762-42dd-4ba0-8ea2-06f812648a5e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:41.823527 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:41.827336 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:41.833149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:41.836410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] PATH_INFO: `/v3/projects/b0d2ae984c084f2e8c6964177fde2678` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:41.847322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0d2ae984c084f2e8c6964177fde2678)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:41.861342 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:41.972161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:42.024059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0d2ae984c084f2e8c6964177fde2678', 'request_id': 'req-1420d422-2fba-45df-9525-bafbde93b5e8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.050722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0d2ae984c084f2e8c6964177fde2678', 'request_id': 'req-1420d422-2fba-45df-9525-bafbde93b5e8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.075085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0d2ae984c084f2e8c6964177fde2678', 'request_id': 'req-1420d422-2fba-45df-9525-bafbde93b5e8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.100675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] The token cache is being invalidate because project b0d2ae984c084f2e8c6964177fde2678 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:42.102373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.104197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.109969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 983/1952] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:41 2022] DELETE /identity/v3/projects/b0d2ae984c084f2e8c6964177fde2678 => generated 0 bytes in 462 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:42.113846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1bb078-e453-4b1b-99db-24118ad30502 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.114839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1bb078-e453-4b1b-99db-24118ad30502 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.115919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1bb078-e453-4b1b-99db-24118ad30502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.189932 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ac1bb078-e453-4b1b-99db-24118ad30502 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:42.191256 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ac1bb078-e453-4b1b-99db-24118ad30502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:42.278617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:42.284018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:42.309949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 970/1953] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:42.319739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58f9e6-df95-4dd7-ba73-8bcbd4777551 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.319941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58f9e6-df95-4dd7-ba73-8bcbd4777551 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.320287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58f9e6-df95-4dd7-ba73-8bcbd4777551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.359789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-cc58f9e6-df95-4dd7-ba73-8bcbd4777551 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:42.361342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-cc58f9e6-df95-4dd7-ba73-8bcbd4777551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:42.381407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 971/1954] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:42.386248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58f9e6-df95-4dd7-ba73-8bcbd4777551 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:42.401423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58f9e6-df95-4dd7-ba73-8bcbd4777551 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:42.408417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1420d422-2fba-45df-9525-bafbde93b5e8 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:42.409584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:42.410216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.410592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.410948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.411720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:42.412035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:42.413645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:42.429757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 972/1955] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 14:10:42.430344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:42.431077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.431433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.431806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] PATH_INFO: `/v3/users/995acbc8292a4335ab13e4928cb37aa8` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.433326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] RBAC: Authorizing `identity:delete_user(user_id=995acbc8292a4335ab13e4928cb37aa8)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:42.434844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:42.435485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:42.452976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ab568c-3a08-4b9f-9fa0-0722b7c11ccd admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:42.463579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3850d697-3f0a-46e5-af95-9d090f016459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:42.464790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3850d697-3f0a-46e5-af95-9d090f016459 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.465330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3850d697-3f0a-46e5-af95-9d090f016459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.466775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3850d697-3f0a-46e5-af95-9d090f016459 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.468196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3850d697-3f0a-46e5-af95-9d090f016459 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:42.470799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3850d697-3f0a-46e5-af95-9d090f016459 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:42.502912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:42.523571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '995acbc8292a4335ab13e4928cb37aa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.529978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 973/1956] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:42 2022] POST /identity/v3/projects => generated 407 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:42.531213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3850d697-3f0a-46e5-af95-9d090f016459 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:42.533172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '995acbc8292a4335ab13e4928cb37aa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.533832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '995acbc8292a4335ab13e4928cb37aa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.534923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '995acbc8292a4335ab13e4928cb37aa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.543089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3cf81384-94c9-41e3-9b92-4b16201e3f21 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '995acbc8292a4335ab13e4928cb37aa8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:42.564406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 984/1957] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:42 2022] DELETE /identity/v3/users/995acbc8292a4335ab13e4928cb37aa8 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:42.568915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe822bd-1d9c-4417-84dd-c436fbce93dc None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.569412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe822bd-1d9c-4417-84dd-c436fbce93dc None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.569682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe822bd-1d9c-4417-84dd-c436fbce93dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.595568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-dfe822bd-1d9c-4417-84dd-c436fbce93dc None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:42.596811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dfe822bd-1d9c-4417-84dd-c436fbce93dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:42.622607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3850d697-3f0a-46e5-af95-9d090f016459 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:42.631472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:42.666319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 985/1958] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:10:42.677685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe822bd-1d9c-4417-84dd-c436fbce93dc None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:42.679169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dfe822bd-1d9c-4417-84dd-c436fbce93dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:42.704370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:42.705134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.705426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.705703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.706417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:42.708380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:42.733747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 974/1959] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:42 2022] POST /identity/v3/users => generated 304 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:10:42.748936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:42.754774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:42.787159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023c154f-0ecd-4d4c-a8bd-bca90676f504 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:42.805040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe822bd-1d9c-4417-84dd-c436fbce93dc None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:42.806246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c9f1e5-72a1-44bb-af48-a39dbb2728e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:42.807073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9f1e5-72a1-44bb-af48-a39dbb2728e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.807406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9f1e5-72a1-44bb-af48-a39dbb2728e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.808122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9f1e5-72a1-44bb-af48-a39dbb2728e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.808757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c9f1e5-72a1-44bb-af48-a39dbb2728e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:42.811166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c9f1e5-72a1-44bb-af48-a39dbb2728e4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:42.828823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 975/1960] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:42 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:42.829060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a143d9-ac7d-4da5-8d61-a2dc802399ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:42.829060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a143d9-ac7d-4da5-8d61-a2dc802399ff service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.829060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a143d9-ac7d-4da5-8d61-a2dc802399ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.829060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a143d9-ac7d-4da5-8d61-a2dc802399ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.836576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c9f1e5-72a1-44bb-af48-a39dbb2728e4 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:42.841618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a143d9-ac7d-4da5-8d61-a2dc802399ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:42.843851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a143d9-ac7d-4da5-8d61-a2dc802399ff service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:42.853232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c9f1e5-72a1-44bb-af48-a39dbb2728e4 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:42.860246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c639a2-8427-48b3-b642-a1e2bffd34f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:42.861209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c639a2-8427-48b3-b642-a1e2bffd34f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.861550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c639a2-8427-48b3-b642-a1e2bffd34f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.864915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c639a2-8427-48b3-b642-a1e2bffd34f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.865077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 986/1961] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:10:42.865178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c639a2-8427-48b3-b642-a1e2bffd34f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:42.865178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c639a2-8427-48b3-b642-a1e2bffd34f7 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:42.886533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 976/1962] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:42 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:42.890648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a143d9-ac7d-4da5-8d61-a2dc802399ff service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:42.903508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a143d9-ac7d-4da5-8d61-a2dc802399ff service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:42.915511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f334084-9e69-4552-8c80-65dc1f00218d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:42.916177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f334084-9e69-4552-8c80-65dc1f00218d admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.916458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f334084-9e69-4552-8c80-65dc1f00218d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.916733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f334084-9e69-4552-8c80-65dc1f00218d admin admin] PATH_INFO: `/v3/projects/7ff80a6a168245108bdb8a11b883fd7d/users/158e5b6544874fa7917093ef5c524847/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:42.942935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f334084-9e69-4552-8c80-65dc1f00218d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ff80a6a168245108bdb8a11b883fd7d, user_id=158e5b6544874fa7917093ef5c524847, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:42.946959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f334084-9e69-4552-8c80-65dc1f00218d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:42.974169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 987/1963] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:42 2022] PUT /identity/v3/projects/7ff80a6a168245108bdb8a11b883fd7d/users/158e5b6544874fa7917093ef5c524847/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:42.980477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010cc96a-2963-45c7-8ccb-4aec72db4f7a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:42.981053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010cc96a-2963-45c7-8ccb-4aec72db4f7a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:42.981795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010cc96a-2963-45c7-8ccb-4aec72db4f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.026929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-010cc96a-2963-45c7-8ccb-4aec72db4f7a None None] MFA Rules not processed for user `158e5b6544874fa7917093ef5c524847`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:43.031395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-010cc96a-2963-45c7-8ccb-4aec72db4f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:43.164898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:43.175726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 977/1964] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:42 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:43.176925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f334084-9e69-4552-8c80-65dc1f00218d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:43.259039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f334084-9e69-4552-8c80-65dc1f00218d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:43.266740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0a7f80-295a-4f7f-b2ad-df839e065aec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:43.267586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7f80-295a-4f7f-b2ad-df839e065aec admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:43.267996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7f80-295a-4f7f-b2ad-df839e065aec admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:43.268384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7f80-295a-4f7f-b2ad-df839e065aec admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.269219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0a7f80-295a-4f7f-b2ad-df839e065aec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:43.271635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0a7f80-295a-4f7f-b2ad-df839e065aec admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:43.319376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 988/1965] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:43 2022] POST /identity/v3/projects => generated 409 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:43.323605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010cc96a-2963-45c7-8ccb-4aec72db4f7a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:43.346939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010cc96a-2963-45c7-8ccb-4aec72db4f7a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:43.421000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c331938-b61d-49e6-bc61-cb3081748858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:43.421713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c331938-b61d-49e6-bc61-cb3081748858 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:43.422075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c331938-b61d-49e6-bc61-cb3081748858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:43.422568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c331938-b61d-49e6-bc61-cb3081748858 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.423295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c331938-b61d-49e6-bc61-cb3081748858 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:43.426228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c331938-b61d-49e6-bc61-cb3081748858 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:43.485214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 978/1966] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:43 2022] POST /identity/v3/users => generated 311 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:10:43.489953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0a7f80-295a-4f7f-b2ad-df839e065aec admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:43.510953 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0a7f80-295a-4f7f-b2ad-df839e065aec admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:43.520876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:43.521489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:43.521772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:43.522045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.522797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:43.524574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:43.542408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 989/1967] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:43 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:43.544534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c331938-b61d-49e6-bc61-cb3081748858 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:43.544680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:43.546839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:43.564448 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c331938-b61d-49e6-bc61-cb3081748858 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:43.568720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c79af1d-baed-42c6-8ffe-545ed6e33028 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:43.575139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:43.576176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:43.576951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38f754b-3f97-485b-aee1-899a9d139fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:43.577082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:43.577082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] PATH_INFO: `/v3/projects/7190d2ccbbe8414cadd765ae26c2946b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.577348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f754b-3f97-485b-aee1-899a9d139fb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:43.578208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f754b-3f97-485b-aee1-899a9d139fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:43.578509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38f754b-3f97-485b-aee1-899a9d139fb2 admin admin] PATH_INFO: `/v3/projects/c6f9a6da864d4670a84a0e712d526c81/users/c634cb83513c4b4a8aff8a97cb1b7fc0/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.579794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7190d2ccbbe8414cadd765ae26c2946b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:43.582768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:43.610969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38f754b-3f97-485b-aee1-899a9d139fb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6f9a6da864d4670a84a0e712d526c81, user_id=c634cb83513c4b4a8aff8a97cb1b7fc0, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:43.613009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38f754b-3f97-485b-aee1-899a9d139fb2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:43.638408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 979/1968] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:43 2022] PUT /identity/v3/projects/c6f9a6da864d4670a84a0e712d526c81/users/c634cb83513c4b4a8aff8a97cb1b7fc0/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:43.639490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38f754b-3f97-485b-aee1-899a9d139fb2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:43.666469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:43.701704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7190d2ccbbe8414cadd765ae26c2946b', 'request_id': 'req-08268fcf-99cb-4f01-9a36-0c2b34753201'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.719520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7190d2ccbbe8414cadd765ae26c2946b', 'request_id': 'req-08268fcf-99cb-4f01-9a36-0c2b34753201'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.732470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7190d2ccbbe8414cadd765ae26c2946b', 'request_id': 'req-08268fcf-99cb-4f01-9a36-0c2b34753201'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.764006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] The token cache is being invalidate because project 7190d2ccbbe8414cadd765ae26c2946b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:43.764581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.767294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.770715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 990/1969] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:43 2022] DELETE /identity/v3/projects/7190d2ccbbe8414cadd765ae26c2946b => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:43.776536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:43.778437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:43.852981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38f754b-3f97-485b-aee1-899a9d139fb2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:43.903479 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08268fcf-99cb-4f01-9a36-0c2b34753201 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:43.910883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:43.912652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:43.912945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:43.916242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] PATH_INFO: `/v3/users/3f75554affb94f0b9f12438ab162aa96` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.918075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f75554affb94f0b9f12438ab162aa96)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:43.920494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:43.938235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:43.939164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:43.939505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:43.939883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.940593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:43.942256 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:43.956601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 980/1970] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:43 2022] GET /identity/v3/roles => generated 1420 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:43.961225 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:43.963137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:43.974915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f75554affb94f0b9f12438ab162aa96'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.975618 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56693f-a0b2-4fdb-96b6-7149c7ad40bb admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:43.981450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b7316b-e997-43f7-bf0c-780dc6fc1380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:43.981721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f75554affb94f0b9f12438ab162aa96'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.982308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7316b-e997-43f7-bf0c-780dc6fc1380 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:43.982420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f75554affb94f0b9f12438ab162aa96'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.982710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7316b-e997-43f7-bf0c-780dc6fc1380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:43.982990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f75554affb94f0b9f12438ab162aa96'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.983347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7316b-e997-43f7-bf0c-780dc6fc1380 admin admin] PATH_INFO: `/v3/projects/c6f9a6da864d4670a84a0e712d526c81/users/c634cb83513c4b4a8aff8a97cb1b7fc0/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:43.986879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b7316b-e997-43f7-bf0c-780dc6fc1380 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6f9a6da864d4670a84a0e712d526c81, user_id=c634cb83513c4b4a8aff8a97cb1b7fc0, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:43.990348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b7316b-e997-43f7-bf0c-780dc6fc1380 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:43.991942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f75554affb94f0b9f12438ab162aa96'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:43.992924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:44.004732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 981/1971] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:43 2022] PUT /identity/v3/projects/c6f9a6da864d4670a84a0e712d526c81/users/c634cb83513c4b4a8aff8a97cb1b7fc0/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:44.013541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 991/1972] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:43 2022] DELETE /identity/v3/users/3f75554affb94f0b9f12438ab162aa96 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:44.015968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81a6b5-cc1c-4b6c-8d24-262c849785a2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.016299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81a6b5-cc1c-4b6c-8d24-262c849785a2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.016613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81a6b5-cc1c-4b6c-8d24-262c849785a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.046992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-3b81a6b5-cc1c-4b6c-8d24-262c849785a2 None None] MFA Rules not processed for user `c634cb83513c4b4a8aff8a97cb1b7fc0`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:44.048356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3b81a6b5-cc1c-4b6c-8d24-262c849785a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:44.133816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 982/1973] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:44 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:44.145956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:44.147337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:44.198723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:44.219569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd922e5-e929-4107-be24-a55fd3024f37 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:44.225200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e5a909-4d4b-485e-b808-12ac688740db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:44.226241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e5a909-4d4b-485e-b808-12ac688740db service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.226241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e5a909-4d4b-485e-b808-12ac688740db service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.226548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e5a909-4d4b-485e-b808-12ac688740db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.238233 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e5a909-4d4b-485e-b808-12ac688740db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:44.239898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e5a909-4d4b-485e-b808-12ac688740db service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:44.254411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 992/1974] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:44 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:44.378164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81a6b5-cc1c-4b6c-8d24-262c849785a2 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:44.379636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3b81a6b5-cc1c-4b6c-8d24-262c849785a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:44.467002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e5a909-4d4b-485e-b808-12ac688740db service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:44.477107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e5a909-4d4b-485e-b808-12ac688740db service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:44.481625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e806c6-6b6b-4f29-a43b-21f861717eaf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:44.482224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e806c6-6b6b-4f29-a43b-21f861717eaf service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.482502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e806c6-6b6b-4f29-a43b-21f861717eaf service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.482776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e806c6-6b6b-4f29-a43b-21f861717eaf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.485179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-79e806c6-6b6b-4f29-a43b-21f861717eaf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:44.485465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81a6b5-cc1c-4b6c-8d24-262c849785a2 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:44.490308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:44.491081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.491457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.491864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] PATH_INFO: `/v3/projects/bba77e8815c44481868cff88657abcd8` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.493099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bba77e8815c44481868cff88657abcd8)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:44.495145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:44.550289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bba77e8815c44481868cff88657abcd8', 'request_id': 'req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:44.551954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e806c6-6b6b-4f29-a43b-21f861717eaf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:44.553332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e806c6-6b6b-4f29-a43b-21f861717eaf service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:44.559325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bba77e8815c44481868cff88657abcd8', 'request_id': 'req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:44.565794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 993/1975] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:44 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:10:44.565948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bba77e8815c44481868cff88657abcd8', 'request_id': 'req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:44.582123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] The token cache is being invalidate because project bba77e8815c44481868cff88657abcd8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:44.582413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:44.583066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c10e2d3a-8f43-4d61-a335-138ccc99d4d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:44.584847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 983/1976] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:44 2022] DELETE /identity/v3/projects/bba77e8815c44481868cff88657abcd8 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:44.589554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8562e4-d32f-46fa-8744-98c91230c049 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.590407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8562e4-d32f-46fa-8744-98c91230c049 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.590407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8562e4-d32f-46fa-8744-98c91230c049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.619384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3d8562e4-d32f-46fa-8744-98c91230c049 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:44.620786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3d8562e4-d32f-46fa-8744-98c91230c049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:44.697722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 994/1977] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:44.706514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27741bcb-131f-4a1c-938e-dd384088ffe9 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.707893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27741bcb-131f-4a1c-938e-dd384088ffe9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.708209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27741bcb-131f-4a1c-938e-dd384088ffe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.732720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-27741bcb-131f-4a1c-938e-dd384088ffe9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:44.734321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-27741bcb-131f-4a1c-938e-dd384088ffe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:44.743363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 984/1978] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:44.747704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8562e4-d32f-46fa-8744-98c91230c049 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:44.757871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8562e4-d32f-46fa-8744-98c91230c049 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:44.762580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:44.763185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.763617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.763931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.765255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:44.765567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:44.772076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:44.781468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 995/1979] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:44.786004 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27741bcb-131f-4a1c-938e-dd384088ffe9 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:44.792615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:44.798202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27741bcb-131f-4a1c-938e-dd384088ffe9 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:44.811948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dccfd3-d9e5-4a7c-8b49-fd04007d1bf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:44.812883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dccfd3-d9e5-4a7c-8b49-fd04007d1bf8 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.813146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dccfd3-d9e5-4a7c-8b49-fd04007d1bf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.813415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dccfd3-d9e5-4a7c-8b49-fd04007d1bf8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.814070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dccfd3-d9e5-4a7c-8b49-fd04007d1bf8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:44.816068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dccfd3-d9e5-4a7c-8b49-fd04007d1bf8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:44.853617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 985/1980] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:44 2022] POST /identity/v3/projects => generated 383 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:44.861518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:44.878877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225d3fea-78cd-4470-ae00-89c452d028d1 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:44.955753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe8285-935d-4af4-9f9e-0b1165cf94f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:44.956692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe8285-935d-4af4-9f9e-0b1165cf94f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:44.956988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe8285-935d-4af4-9f9e-0b1165cf94f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:44.957274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe8285-935d-4af4-9f9e-0b1165cf94f3 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:44.957946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe8285-935d-4af4-9f9e-0b1165cf94f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:44.959923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe8285-935d-4af4-9f9e-0b1165cf94f3 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:44.971037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:45.005669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dccfd3-d9e5-4a7c-8b49-fd04007d1bf8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.006728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 996/1981] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:44 2022] POST /identity/v3/users => generated 292 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:45.020896 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dccfd3-d9e5-4a7c-8b49-fd04007d1bf8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.026862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4961a2f0-22c8-4dc5-839d-58288ca4f449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.028397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4961a2f0-22c8-4dc5-839d-58288ca4f449 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.028692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4961a2f0-22c8-4dc5-839d-58288ca4f449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.028948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4961a2f0-22c8-4dc5-839d-58288ca4f449 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.029660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4961a2f0-22c8-4dc5-839d-58288ca4f449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.031869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4961a2f0-22c8-4dc5-839d-58288ca4f449 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.043592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 986/1982] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:45 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:45.049248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe8285-935d-4af4-9f9e-0b1165cf94f3 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.064642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe8285-935d-4af4-9f9e-0b1165cf94f3 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.069447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959130f5-cb45-4343-b53e-9593f09cfdf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.070054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959130f5-cb45-4343-b53e-9593f09cfdf3 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.070386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959130f5-cb45-4343-b53e-9593f09cfdf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.070663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959130f5-cb45-4343-b53e-9593f09cfdf3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.071312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959130f5-cb45-4343-b53e-9593f09cfdf3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.072940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959130f5-cb45-4343-b53e-9593f09cfdf3 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.084341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 997/1983] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:45 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:45.088614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4961a2f0-22c8-4dc5-839d-58288ca4f449 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.099087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4961a2f0-22c8-4dc5-839d-58288ca4f449 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.103855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86aef1a0-3185-4220-9ca4-e8a82ab73dd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.104516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aef1a0-3185-4220-9ca4-e8a82ab73dd7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.104797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aef1a0-3185-4220-9ca4-e8a82ab73dd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.105073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aef1a0-3185-4220-9ca4-e8a82ab73dd7 admin admin] PATH_INFO: `/v3/projects/16d484101cff4a7581ccb78699edc579/users/01f2f955a6594c4d84103154d7ec57bf/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.120836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86aef1a0-3185-4220-9ca4-e8a82ab73dd7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16d484101cff4a7581ccb78699edc579, user_id=01f2f955a6594c4d84103154d7ec57bf, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.123489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86aef1a0-3185-4220-9ca4-e8a82ab73dd7 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.137370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 987/1984] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:45 2022] PUT /identity/v3/projects/16d484101cff4a7581ccb78699edc579/users/01f2f955a6594c4d84103154d7ec57bf/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:45.147638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b09ff99-0feb-4dd9-b2f2-a585752bb6cb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.147955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b09ff99-0feb-4dd9-b2f2-a585752bb6cb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.148346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b09ff99-0feb-4dd9-b2f2-a585752bb6cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.177148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-0b09ff99-0feb-4dd9-b2f2-a585752bb6cb None None] MFA Rules not processed for user `01f2f955a6594c4d84103154d7ec57bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:45.178358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0b09ff99-0feb-4dd9-b2f2-a585752bb6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:45.243359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 998/1985] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:45.247937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86aef1a0-3185-4220-9ca4-e8a82ab73dd7 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.309230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86aef1a0-3185-4220-9ca4-e8a82ab73dd7 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.314241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f35088-2b09-4026-8245-548dd422899a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.314850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f35088-2b09-4026-8245-548dd422899a admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.315126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f35088-2b09-4026-8245-548dd422899a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.315395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f35088-2b09-4026-8245-548dd422899a admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.316059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f35088-2b09-4026-8245-548dd422899a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.317975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f35088-2b09-4026-8245-548dd422899a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.345321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 988/1986] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:45 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:10:45.349348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b09ff99-0feb-4dd9-b2f2-a585752bb6cb None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.360353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b09ff99-0feb-4dd9-b2f2-a585752bb6cb None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.415922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4bc6f9-d67b-4043-a62a-311c34afbf6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.416529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4bc6f9-d67b-4043-a62a-311c34afbf6c admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.416813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4bc6f9-d67b-4043-a62a-311c34afbf6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.417084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4bc6f9-d67b-4043-a62a-311c34afbf6c admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.417708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4bc6f9-d67b-4043-a62a-311c34afbf6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.419619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4bc6f9-d67b-4043-a62a-311c34afbf6c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.441350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 999/1987] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:45 2022] POST /identity/v3/users => generated 298 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:45.444893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f35088-2b09-4026-8245-548dd422899a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.456686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f35088-2b09-4026-8245-548dd422899a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.461552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1643ad64-fdda-41c8-9cad-f44c722db59e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.462160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1643ad64-fdda-41c8-9cad-f44c722db59e admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.462467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1643ad64-fdda-41c8-9cad-f44c722db59e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.462741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1643ad64-fdda-41c8-9cad-f44c722db59e admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.463386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1643ad64-fdda-41c8-9cad-f44c722db59e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.465060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1643ad64-fdda-41c8-9cad-f44c722db59e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.477071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 989/1988] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:45.481189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4bc6f9-d67b-4043-a62a-311c34afbf6c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.491397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4bc6f9-d67b-4043-a62a-311c34afbf6c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.495978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b396a4-0f17-4a83-b312-6804bad3eada admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.496632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b396a4-0f17-4a83-b312-6804bad3eada admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.496915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b396a4-0f17-4a83-b312-6804bad3eada admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.497193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b396a4-0f17-4a83-b312-6804bad3eada admin admin] PATH_INFO: `/v3/projects/2d643db6f4e44f37976c2d8ca13b23cb/users/401fdd290a984ce387fa084a72fd7c0c/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.512586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b396a4-0f17-4a83-b312-6804bad3eada admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d643db6f4e44f37976c2d8ca13b23cb, user_id=401fdd290a984ce387fa084a72fd7c0c, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.515838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b396a4-0f17-4a83-b312-6804bad3eada admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.529162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1000/1989] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:45 2022] PUT /identity/v3/projects/2d643db6f4e44f37976c2d8ca13b23cb/users/401fdd290a984ce387fa084a72fd7c0c/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:45.533723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1643ad64-fdda-41c8-9cad-f44c722db59e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.543674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1643ad64-fdda-41c8-9cad-f44c722db59e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.603066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3496cc5-51fe-4905-b183-551d2e33ffc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.603826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3496cc5-51fe-4905-b183-551d2e33ffc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.604181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3496cc5-51fe-4905-b183-551d2e33ffc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.604525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3496cc5-51fe-4905-b183-551d2e33ffc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.605328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3496cc5-51fe-4905-b183-551d2e33ffc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.606979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3496cc5-51fe-4905-b183-551d2e33ffc0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.617832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 990/1990] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:45 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:45.621789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b396a4-0f17-4a83-b312-6804bad3eada admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.632674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b396a4-0f17-4a83-b312-6804bad3eada admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.637197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.637880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.638152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.638424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] PATH_INFO: `/v3/projects/2d643db6f4e44f37976c2d8ca13b23cb/users/401fdd290a984ce387fa084a72fd7c0c/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.640659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d643db6f4e44f37976c2d8ca13b23cb, user_id=401fdd290a984ce387fa084a72fd7c0c, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.643478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.653636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1001/1991] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:45 2022] PUT /identity/v3/projects/2d643db6f4e44f37976c2d8ca13b23cb/users/401fdd290a984ce387fa084a72fd7c0c/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:45.665575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b932170-fa23-4db2-9466-56b2b4cbe866 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.665991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b932170-fa23-4db2-9466-56b2b4cbe866 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.666271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b932170-fa23-4db2-9466-56b2b4cbe866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.695518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2b932170-fa23-4db2-9466-56b2b4cbe866 None None] MFA Rules not processed for user `401fdd290a984ce387fa084a72fd7c0c`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:45.696777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2b932170-fa23-4db2-9466-56b2b4cbe866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:45.700745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:45.772238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 991/1992] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:45 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:45.784871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.786558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:45.815242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:45.847110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dae3d9-0c71-4562-a136-48918993e708 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.850770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4abbf4-67c8-433f-a1ea-0e7e93d7cac7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.851374 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4abbf4-67c8-433f-a1ea-0e7e93d7cac7 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.851650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4abbf4-67c8-433f-a1ea-0e7e93d7cac7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.851945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4abbf4-67c8-433f-a1ea-0e7e93d7cac7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:45.861473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4abbf4-67c8-433f-a1ea-0e7e93d7cac7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:45.862888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4abbf4-67c8-433f-a1ea-0e7e93d7cac7 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:45.872167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1002/1993] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:45 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:45.947143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b932170-fa23-4db2-9466-56b2b4cbe866 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:45.948997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2b932170-fa23-4db2-9466-56b2b4cbe866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:45.963427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b932170-fa23-4db2-9466-56b2b4cbe866 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:45.967558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0269efd0-6e15-4507-ab6d-b245afea7581 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:45.970891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269efd0-6e15-4507-ab6d-b245afea7581 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:45.971175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269efd0-6e15-4507-ab6d-b245afea7581 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:45.971444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269efd0-6e15-4507-ab6d-b245afea7581 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:46.034818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0269efd0-6e15-4507-ab6d-b245afea7581 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:46.036192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0269efd0-6e15-4507-ab6d-b245afea7581 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:46.044902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 992/1994] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:45 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:46.351659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4abbf4-67c8-433f-a1ea-0e7e93d7cac7 service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:46.417204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4abbf4-67c8-433f-a1ea-0e7e93d7cac7 service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:46.421777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:46.422408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:46.422688 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:46.422965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] PATH_INFO: `/v3/users/01f2f955a6594c4d84103154d7ec57bf` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:46.424114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] RBAC: Authorizing `identity:delete_user(user_id=01f2f955a6594c4d84103154d7ec57bf)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:46.426043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:46.473709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01f2f955a6594c4d84103154d7ec57bf'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:46.481124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01f2f955a6594c4d84103154d7ec57bf'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:46.482011 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01f2f955a6594c4d84103154d7ec57bf'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:46.482614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01f2f955a6594c4d84103154d7ec57bf'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:46.490043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01f2f955a6594c4d84103154d7ec57bf'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:46.508241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1003/1995] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:46 2022] DELETE /identity/v3/users/01f2f955a6594c4d84103154d7ec57bf => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:46.513216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126009e1-1fcd-4c9f-9b21-75d93d86b587 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:46.513701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126009e1-1fcd-4c9f-9b21-75d93d86b587 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:46.513790 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126009e1-1fcd-4c9f-9b21-75d93d86b587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:46.533267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-126009e1-1fcd-4c9f-9b21-75d93d86b587 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:46.534438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-126009e1-1fcd-4c9f-9b21-75d93d86b587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:46.596901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 993/1996] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:46.606501 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:46.607973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:46.661260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:46.667505 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3aa6f9-6199-43e7-9c19-bba051852055 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:46.672608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc76968c-2796-44b8-a324-b0fe73f8566d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:46.673225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc76968c-2796-44b8-a324-b0fe73f8566d service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:46.673504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc76968c-2796-44b8-a324-b0fe73f8566d service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:46.673778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc76968c-2796-44b8-a324-b0fe73f8566d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:46.683483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc76968c-2796-44b8-a324-b0fe73f8566d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:46.684962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc76968c-2796-44b8-a324-b0fe73f8566d service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:46.695389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1004/1997] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:47.060079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126009e1-1fcd-4c9f-9b21-75d93d86b587 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:47.061650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-126009e1-1fcd-4c9f-9b21-75d93d86b587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:47.076827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126009e1-1fcd-4c9f-9b21-75d93d86b587 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:47.082312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:47.083037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:47.083255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:47.083527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] PATH_INFO: `/v3/projects/16d484101cff4a7581ccb78699edc579` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:47.084875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] RBAC: Authorizing `identity:delete_project(project_id=16d484101cff4a7581ccb78699edc579)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:47.086752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:47.123890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:47.134389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16d484101cff4a7581ccb78699edc579', 'request_id': 'req-601c0add-b741-480e-9e82-2234c8ac266b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.142638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16d484101cff4a7581ccb78699edc579', 'request_id': 'req-601c0add-b741-480e-9e82-2234c8ac266b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.148419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16d484101cff4a7581ccb78699edc579', 'request_id': 'req-601c0add-b741-480e-9e82-2234c8ac266b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.159957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] The token cache is being invalidate because project 16d484101cff4a7581ccb78699edc579 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:47.160245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.160872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.162367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 994/1998] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:47 2022] DELETE /identity/v3/projects/16d484101cff4a7581ccb78699edc579 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:47.166385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc76968c-2796-44b8-a324-b0fe73f8566d service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:47.176210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc76968c-2796-44b8-a324-b0fe73f8566d service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:47.234085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:47.234720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:47.235004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:47.235284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] PATH_INFO: `/v3/users/401fdd290a984ce387fa084a72fd7c0c` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:47.236449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] RBAC: Authorizing `identity:delete_user(user_id=401fdd290a984ce387fa084a72fd7c0c)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:47.238389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:47.284115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '401fdd290a984ce387fa084a72fd7c0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.291290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '401fdd290a984ce387fa084a72fd7c0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.291942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '401fdd290a984ce387fa084a72fd7c0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.292550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '401fdd290a984ce387fa084a72fd7c0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.300007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5e66c10e-4d8a-4a21-a158-e218be8f8f94 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '401fdd290a984ce387fa084a72fd7c0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.322263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1005/1999] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:10:47 2022] DELETE /identity/v3/users/401fdd290a984ce387fa084a72fd7c0c => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:47.680875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:47.682379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:47.697353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601c0add-b741-480e-9e82-2234c8ac266b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:47.755000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:47.755633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:47.755944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:47.756221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] PATH_INFO: `/v3/projects/2d643db6f4e44f37976c2d8ca13b23cb` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:47.757323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d643db6f4e44f37976c2d8ca13b23cb)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:47.759237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:47.806477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d643db6f4e44f37976c2d8ca13b23cb', 'request_id': 'req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.813360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d643db6f4e44f37976c2d8ca13b23cb', 'request_id': 'req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.818970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d643db6f4e44f37976c2d8ca13b23cb', 'request_id': 'req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.834703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] The token cache is being invalidate because project 2d643db6f4e44f37976c2d8ca13b23cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:10:47.835220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.836297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-dbbd1335-3f3a-4a1a-a7f8-d45f54bda0f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:10:47.838842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 995/2000] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:10:47 2022] DELETE /identity/v3/projects/2d643db6f4e44f37976c2d8ca13b23cb => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:10:47.847919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75390eb-ee0c-4e33-9bb7-d55d67f26fae None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:47.848375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75390eb-ee0c-4e33-9bb7-d55d67f26fae None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:47.848457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75390eb-ee0c-4e33-9bb7-d55d67f26fae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:47.867608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a75390eb-ee0c-4e33-9bb7-d55d67f26fae None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:47.868785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a75390eb-ee0c-4e33-9bb7-d55d67f26fae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:47.945731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1006/2001] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:47.954245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11555fed-6d16-472c-b254-9967aa273610 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:47.954535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11555fed-6d16-472c-b254-9967aa273610 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:47.954816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11555fed-6d16-472c-b254-9967aa273610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:47.975213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-11555fed-6d16-472c-b254-9967aa273610 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:47.976474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-11555fed-6d16-472c-b254-9967aa273610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:47.984740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 996/2002] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:47.988690 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75390eb-ee0c-4e33-9bb7-d55d67f26fae None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:48.000419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75390eb-ee0c-4e33-9bb7-d55d67f26fae None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:48.005085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:48.005698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:48.005975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:48.006244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:48.006893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:10:48.007195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:48.008836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:48.011448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:48.028686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11555fed-6d16-472c-b254-9967aa273610 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:48.035873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1007/2003] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:10:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:10:48.043774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11555fed-6d16-472c-b254-9967aa273610 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:48.052541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a777bf7-274f-4a10-8125-044b8c5ecc09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:48.053166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a777bf7-274f-4a10-8125-044b8c5ecc09 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:48.053450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a777bf7-274f-4a10-8125-044b8c5ecc09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:48.053719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a777bf7-274f-4a10-8125-044b8c5ecc09 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:48.054359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a777bf7-274f-4a10-8125-044b8c5ecc09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:48.056394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a777bf7-274f-4a10-8125-044b8c5ecc09 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:48.105494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 997/2004] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:10:48 2022] POST /identity/v3/projects => generated 377 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:10:48.107519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:48.126161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8181bd9-43bb-4bf3-ba14-0212a2d5024a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:48.185219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2e90d2-0feb-453a-a43f-47f2cf4bf471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:48.187037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2e90d2-0feb-453a-a43f-47f2cf4bf471 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:48.187406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2e90d2-0feb-453a-a43f-47f2cf4bf471 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:48.187841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2e90d2-0feb-453a-a43f-47f2cf4bf471 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:48.188499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2e90d2-0feb-453a-a43f-47f2cf4bf471 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:48.190414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2e90d2-0feb-453a-a43f-47f2cf4bf471 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:48.211504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1008/2005] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:10:48 2022] POST /identity/v3/users => generated 289 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:10:48.215521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a777bf7-274f-4a10-8125-044b8c5ecc09 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:48.226268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a777bf7-274f-4a10-8125-044b8c5ecc09 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:48.230886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f9fc8b-8bf6-4778-a849-51e48844c30a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:48.231479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f9fc8b-8bf6-4778-a849-51e48844c30a admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:48.231797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f9fc8b-8bf6-4778-a849-51e48844c30a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:48.232071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f9fc8b-8bf6-4778-a849-51e48844c30a admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:48.232716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f9fc8b-8bf6-4778-a849-51e48844c30a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:48.234826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f9fc8b-8bf6-4778-a849-51e48844c30a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:48.244675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 998/2006] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:48.248671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2e90d2-0feb-453a-a43f-47f2cf4bf471 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:48.259123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2e90d2-0feb-453a-a43f-47f2cf4bf471 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:48.263672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbc02da-f252-478d-9b9b-66d139ba3c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:48.264302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbc02da-f252-478d-9b9b-66d139ba3c8e admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:48.264586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbc02da-f252-478d-9b9b-66d139ba3c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:48.264864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbc02da-f252-478d-9b9b-66d139ba3c8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:48.265509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbc02da-f252-478d-9b9b-66d139ba3c8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:48.267102 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbc02da-f252-478d-9b9b-66d139ba3c8e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:48.277532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1009/2007] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:10:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:10:48.281520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f9fc8b-8bf6-4778-a849-51e48844c30a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:48.291431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f9fc8b-8bf6-4778-a849-51e48844c30a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:48.296771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:48.297691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:48.297945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:48.298340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] PATH_INFO: `/v3/projects/57a2d639abb946318d3ff8fd4924f977/users/f6c928f94dfe400fa48e6d4e5bc9638c/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:48.310466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:48.315956 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] RBAC: Authorizing `identity:create_grant(project_id=57a2d639abb946318d3ff8fd4924f977, user_id=f6c928f94dfe400fa48e6d4e5bc9638c, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:48.319856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:48.332906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 999/2008] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:10:48 2022] PUT /identity/v3/projects/57a2d639abb946318d3ff8fd4924f977/users/f6c928f94dfe400fa48e6d4e5bc9638c/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:10:48.343363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ce389-5f5a-46f7-8621-faadfb233acd None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:48.343649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ce389-5f5a-46f7-8621-faadfb233acd None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:48.343963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776ce389-5f5a-46f7-8621-faadfb233acd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:48.371347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-776ce389-5f5a-46f7-8621-faadfb233acd None None] MFA Rules not processed for user `f6c928f94dfe400fa48e6d4e5bc9638c`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:10:48.372585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-776ce389-5f5a-46f7-8621-faadfb233acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:48.441223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1010/2009] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:10:48 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:10:48.452689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:10:48.454500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:10:48.511227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479264d2-635a-4642-9b5e-09bb492c09af admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:10:48.514926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43fd53a-7368-4d7d-8b31-d36145c42ede service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '54db2e6805fd40f79c03eb3b2319f2b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:10:48.515746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43fd53a-7368-4d7d-8b31-d36145c42ede service glance] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:10:48.516033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43fd53a-7368-4d7d-8b31-d36145c42ede service glance] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:10:48.516313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43fd53a-7368-4d7d-8b31-d36145c42ede service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:10:48.524505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43fd53a-7368-4d7d-8b31-d36145c42ede service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:10:48.525915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43fd53a-7368-4d7d-8b31-d36145c42ede service glance] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:10:48.535099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1000/2010] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:10:48 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:10:58.438263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=68,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:10:58.530737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:03.939867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776ce389-5f5a-46f7-8621-faadfb233acd None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:03.941316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-776ce389-5f5a-46f7-8621-faadfb233acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:04.011948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776ce389-5f5a-46f7-8621-faadfb233acd None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:04.018669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:04.019333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:04.019627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:04.019940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] PATH_INFO: `/v3/users/48083c132cf94804a2ab5732f1e3e825` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:04.021123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=48083c132cf94804a2ab5732f1e3e825)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:04.023160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:04.075875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48083c132cf94804a2ab5732f1e3e825'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.085113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48083c132cf94804a2ab5732f1e3e825'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.085877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48083c132cf94804a2ab5732f1e3e825'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.086592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48083c132cf94804a2ab5732f1e3e825'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.097218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48083c132cf94804a2ab5732f1e3e825'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.120984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1011/2011] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:03 2022] DELETE /identity/v3/users/48083c132cf94804a2ab5732f1e3e825 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:04.125611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f1231-2cca-4eae-9f3a-fd64983330d9 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:04.126105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f1231-2cca-4eae-9f3a-fd64983330d9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:04.126223 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f1231-2cca-4eae-9f3a-fd64983330d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:04.145525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d43f1231-2cca-4eae-9f3a-fd64983330d9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:04.146741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d43f1231-2cca-4eae-9f3a-fd64983330d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:04.208620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1001/2012] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:04.219429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:04.221013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:04.279076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea6acf0-09ff-44d9-a810-76d60aa71bb1 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:04.283651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a241847f-c5ee-43da-af3e-5c21e60451bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:04.284366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a241847f-c5ee-43da-af3e-5c21e60451bf service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:04.284651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a241847f-c5ee-43da-af3e-5c21e60451bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:04.284927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a241847f-c5ee-43da-af3e-5c21e60451bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:04.294472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a241847f-c5ee-43da-af3e-5c21e60451bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:04.295999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a241847f-c5ee-43da-af3e-5c21e60451bf service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:04.308790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1012/2013] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:04.600083 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43f1231-2cca-4eae-9f3a-fd64983330d9 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:04.601702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d43f1231-2cca-4eae-9f3a-fd64983330d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:04.625605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43f1231-2cca-4eae-9f3a-fd64983330d9 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:04.638607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:04.639222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:04.639565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:04.640395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] PATH_INFO: `/v3/projects/03facb46dcfc4545a87e5980260048ea` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:04.643735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] RBAC: Authorizing `identity:delete_project(project_id=03facb46dcfc4545a87e5980260048ea)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:04.644514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:04.705855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03facb46dcfc4545a87e5980260048ea', 'request_id': 'req-d103e26a-fe05-4057-a8f9-46d6cd390952'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.716668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03facb46dcfc4545a87e5980260048ea', 'request_id': 'req-d103e26a-fe05-4057-a8f9-46d6cd390952'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.725072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03facb46dcfc4545a87e5980260048ea', 'request_id': 'req-d103e26a-fe05-4057-a8f9-46d6cd390952'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.740416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] The token cache is being invalidate because project 03facb46dcfc4545a87e5980260048ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:04.740850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.741471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.743120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1002/2014] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:04 2022] DELETE /identity/v3/projects/03facb46dcfc4545a87e5980260048ea => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:04.747492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a241847f-c5ee-43da-af3e-5c21e60451bf service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:04.759545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a241847f-c5ee-43da-af3e-5c21e60451bf service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:04.783807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:04.852166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:04.852824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:04.853186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:04.853513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] PATH_INFO: `/v3/users/ce6a69b266c64952b94540f3a70e244f` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:04.855059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce6a69b266c64952b94540f3a70e244f)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:04.857363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:04.944269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce6a69b266c64952b94540f3a70e244f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.960653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce6a69b266c64952b94540f3a70e244f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.961325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce6a69b266c64952b94540f3a70e244f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.961937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce6a69b266c64952b94540f3a70e244f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.969958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce6a69b266c64952b94540f3a70e244f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:04.993628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1013/2015] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:04 2022] DELETE /identity/v3/users/ce6a69b266c64952b94540f3a70e244f => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:05.625556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:05.628078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:05.651873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d103e26a-fe05-4057-a8f9-46d6cd390952 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:05.711601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:05.712330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:05.712603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:05.712892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] PATH_INFO: `/v3/users/158e5b6544874fa7917093ef5c524847` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:05.714266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=158e5b6544874fa7917093ef5c524847)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:05.716419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:05.745450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:05.746924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:05.751023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:05.761264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e518f-a486-4bc1-ab0b-aca626fd4cac admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:05.766810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:05.767450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:05.767773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:05.768196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] PATH_INFO: `/v3/projects/1496d6f83bb64c2b9013f2695d24e426` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:05.770162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1496d6f83bb64c2b9013f2695d24e426)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:05.772371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:05.774640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '158e5b6544874fa7917093ef5c524847'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.782878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '158e5b6544874fa7917093ef5c524847'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.783988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '158e5b6544874fa7917093ef5c524847'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.784937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '158e5b6544874fa7917093ef5c524847'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.794037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-83a386bb-47e0-415a-a3fe-38c3d9f0a3c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '158e5b6544874fa7917093ef5c524847'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.816272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1003/2016] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:05 2022] DELETE /identity/v3/users/158e5b6544874fa7917093ef5c524847 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:05.820681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbaed06-2b67-469c-8378-22b1ce208dd6 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:05.821045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbaed06-2b67-469c-8378-22b1ce208dd6 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:05.821365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbaed06-2b67-469c-8378-22b1ce208dd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:05.839493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1496d6f83bb64c2b9013f2695d24e426', 'request_id': 'req-f164860d-4321-49bd-b4f6-dcece9e1dd18'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.840875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-1cbaed06-2b67-469c-8378-22b1ce208dd6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:05.842164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1cbaed06-2b67-469c-8378-22b1ce208dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:05.849987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1496d6f83bb64c2b9013f2695d24e426', 'request_id': 'req-f164860d-4321-49bd-b4f6-dcece9e1dd18'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.861091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1496d6f83bb64c2b9013f2695d24e426', 'request_id': 'req-f164860d-4321-49bd-b4f6-dcece9e1dd18'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.874837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] The token cache is being invalidate because project 1496d6f83bb64c2b9013f2695d24e426 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:05.875022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.875678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f164860d-4321-49bd-b4f6-dcece9e1dd18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:05.877540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1014/2017] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:05 2022] DELETE /identity/v3/projects/1496d6f83bb64c2b9013f2695d24e426 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:11:05.882226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec6304b-af75-4d78-8742-766de2f9c7ec None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:05.882514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec6304b-af75-4d78-8742-766de2f9c7ec None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:05.882796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec6304b-af75-4d78-8742-766de2f9c7ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:05.903322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9ec6304b-af75-4d78-8742-766de2f9c7ec None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:05.904618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9ec6304b-af75-4d78-8742-766de2f9c7ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:05.916435 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1004/2018] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:05.927831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbaed06-2b67-469c-8378-22b1ce208dd6 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:05.929321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1cbaed06-2b67-469c-8378-22b1ce208dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:05.932143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:06.004257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1015/2019] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:06.008553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b5e342-79fe-4dd5-ad42-5979a5da8e6e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.009013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b5e342-79fe-4dd5-ad42-5979a5da8e6e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.009294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b5e342-79fe-4dd5-ad42-5979a5da8e6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.044872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-00b5e342-79fe-4dd5-ad42-5979a5da8e6e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:06.047791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-00b5e342-79fe-4dd5-ad42-5979a5da8e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:06.056598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbaed06-2b67-469c-8378-22b1ce208dd6 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.056851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1016/2020] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:06.062769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b5e342-79fe-4dd5-ad42-5979a5da8e6e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.068822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba6f3bd-69da-48cd-af66-db0021c6c08b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.069258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba6f3bd-69da-48cd-af66-db0021c6c08b service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.069718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba6f3bd-69da-48cd-af66-db0021c6c08b service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.070088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba6f3bd-69da-48cd-af66-db0021c6c08b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.078604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b5e342-79fe-4dd5-ad42-5979a5da8e6e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.085718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba6f3bd-69da-48cd-af66-db0021c6c08b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.086694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.087433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba6f3bd-69da-48cd-af66-db0021c6c08b service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.088132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.088711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.089148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.090196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:06.090506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.092402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.106753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1005/2021] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:06.107761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1017/2022] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:11:06.113184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba6f3bd-69da-48cd-af66-db0021c6c08b service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.123261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba6f3bd-69da-48cd-af66-db0021c6c08b service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.131047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6363337-9edf-4282-83ce-6e3d2f75230c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.131968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6363337-9edf-4282-83ce-6e3d2f75230c admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.132472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6363337-9edf-4282-83ce-6e3d2f75230c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.132690 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6363337-9edf-4282-83ce-6e3d2f75230c admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.133298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6363337-9edf-4282-83ce-6e3d2f75230c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.135227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6363337-9edf-4282-83ce-6e3d2f75230c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.165679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1006/2023] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:06 2022] POST /identity/v3/projects => generated 357 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:06.169821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.180400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dacb931-3499-4cf7-b004-cad853620512 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.254887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53111cde-058c-4ed8-a4da-efe90d5d701b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.255500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53111cde-058c-4ed8-a4da-efe90d5d701b admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.255818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53111cde-058c-4ed8-a4da-efe90d5d701b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.256097 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53111cde-058c-4ed8-a4da-efe90d5d701b admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.256742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53111cde-058c-4ed8-a4da-efe90d5d701b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.258675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53111cde-058c-4ed8-a4da-efe90d5d701b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.287513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1018/2024] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:06 2022] POST /identity/v3/users => generated 279 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:06.291406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6363337-9edf-4282-83ce-6e3d2f75230c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.318003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6363337-9edf-4282-83ce-6e3d2f75230c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.323375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c664f557-c113-4366-85f2-84256f9bce4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.324006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c664f557-c113-4366-85f2-84256f9bce4d admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.324284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c664f557-c113-4366-85f2-84256f9bce4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.325059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c664f557-c113-4366-85f2-84256f9bce4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.325258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c664f557-c113-4366-85f2-84256f9bce4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.327438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c664f557-c113-4366-85f2-84256f9bce4d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.349135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1007/2025] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:06 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:06.353466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53111cde-058c-4ed8-a4da-efe90d5d701b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.364872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53111cde-058c-4ed8-a4da-efe90d5d701b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.370388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b56170-8109-447c-a3f0-164b6d1b0485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.371020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b56170-8109-447c-a3f0-164b6d1b0485 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.371306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b56170-8109-447c-a3f0-164b6d1b0485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.371584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b56170-8109-447c-a3f0-164b6d1b0485 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.372290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b56170-8109-447c-a3f0-164b6d1b0485 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.373921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b56170-8109-447c-a3f0-164b6d1b0485 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.387793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1019/2026] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:06 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:06.390196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c664f557-c113-4366-85f2-84256f9bce4d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.402514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c664f557-c113-4366-85f2-84256f9bce4d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.408077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468d932-6690-43eb-bc0d-d679634107fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.408757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468d932-6690-43eb-bc0d-d679634107fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.409037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468d932-6690-43eb-bc0d-d679634107fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.409317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468d932-6690-43eb-bc0d-d679634107fe admin admin] PATH_INFO: `/v3/projects/8e6d672c90fc4025a094d90f76c39e2c/users/7b9670eefd0c49d38a3ff91505ff7bc7/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.421042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:06.426478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e468d932-6690-43eb-bc0d-d679634107fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e6d672c90fc4025a094d90f76c39e2c, user_id=7b9670eefd0c49d38a3ff91505ff7bc7, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.430016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e468d932-6690-43eb-bc0d-d679634107fe admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.458766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1008/2027] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:06 2022] PUT /identity/v3/projects/8e6d672c90fc4025a094d90f76c39e2c/users/7b9670eefd0c49d38a3ff91505ff7bc7/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:06.460743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26476518-a24e-4b94-b37f-4f3dcf790313 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.461027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26476518-a24e-4b94-b37f-4f3dcf790313 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.461340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26476518-a24e-4b94-b37f-4f3dcf790313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.501184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-26476518-a24e-4b94-b37f-4f3dcf790313 None None] MFA Rules not processed for user `7b9670eefd0c49d38a3ff91505ff7bc7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:06.502433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-26476518-a24e-4b94-b37f-4f3dcf790313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:06.578734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1020/2028] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:06 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:06.583405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468d932-6690-43eb-bc0d-d679634107fe admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.687878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468d932-6690-43eb-bc0d-d679634107fe admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.695991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dada8c51-49ba-40b3-8ade-97b966b8b701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.698229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26476518-a24e-4b94-b37f-4f3dcf790313 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.700021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-26476518-a24e-4b94-b37f-4f3dcf790313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:06.704424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dada8c51-49ba-40b3-8ade-97b966b8b701 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.704703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dada8c51-49ba-40b3-8ade-97b966b8b701 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.706245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dada8c51-49ba-40b3-8ade-97b966b8b701 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.707226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dada8c51-49ba-40b3-8ade-97b966b8b701 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.709334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dada8c51-49ba-40b3-8ade-97b966b8b701 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.719760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26476518-a24e-4b94-b37f-4f3dcf790313 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.725253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.726023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.727227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.727601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] PATH_INFO: `/v3/projects/7ff80a6a168245108bdb8a11b883fd7d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.729238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ff80a6a168245108bdb8a11b883fd7d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.731456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.759004 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1009/2029] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:06 2022] POST /identity/v3/projects => generated 357 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:06.767724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:06.772062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dada8c51-49ba-40b3-8ade-97b966b8b701 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.833125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ff80a6a168245108bdb8a11b883fd7d', 'request_id': 'req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:06.863376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ff80a6a168245108bdb8a11b883fd7d', 'request_id': 'req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:06.871055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ff80a6a168245108bdb8a11b883fd7d', 'request_id': 'req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:06.889072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] The token cache is being invalidate because project 7ff80a6a168245108bdb8a11b883fd7d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:06.889703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:06.892331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:06.895259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1021/2030] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:06 2022] DELETE /identity/v3/projects/7ff80a6a168245108bdb8a11b883fd7d => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:11:06.901839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:06.903230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:06.917829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dada8c51-49ba-40b3-8ade-97b966b8b701 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.985526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4beba5-de5c-45c9-86f1-bc235bbf7e17 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:06.990663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.991277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.991559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.991863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.992501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.992618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:06.993302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:06.993674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:06.993997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] PATH_INFO: `/v3/users/c634cb83513c4b4a8aff8a97cb1b7fc0` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:06.994406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:06.995198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c634cb83513c4b4a8aff8a97cb1b7fc0)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:06.997230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:07.019294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1010/2031] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:06 2022] POST /identity/v3/users => generated 285 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:07.022560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:07.025567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:07.039116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913014f0-507d-4497-936b-03f673f88653 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:07.044738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb32df6-c6d8-4907-873f-0579b45ef6b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:07.045358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb32df6-c6d8-4907-873f-0579b45ef6b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.046416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb32df6-c6d8-4907-873f-0579b45ef6b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.046699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb32df6-c6d8-4907-873f-0579b45ef6b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.048515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb32df6-c6d8-4907-873f-0579b45ef6b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:07.049290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb32df6-c6d8-4907-873f-0579b45ef6b4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:07.056818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:07.060832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c634cb83513c4b4a8aff8a97cb1b7fc0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.064296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1011/2032] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:07 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:11:07.069618 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb32df6-c6d8-4907-873f-0579b45ef6b4 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:07.076689 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c634cb83513c4b4a8aff8a97cb1b7fc0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.078415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c634cb83513c4b4a8aff8a97cb1b7fc0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.079162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c634cb83513c4b4a8aff8a97cb1b7fc0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.088977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c634cb83513c4b4a8aff8a97cb1b7fc0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.093679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb32df6-c6d8-4907-873f-0579b45ef6b4 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:07.100459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d8dbb-9e22-403a-b6f9-d6278c863e94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:07.101210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d8dbb-9e22-403a-b6f9-d6278c863e94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.101566 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d8dbb-9e22-403a-b6f9-d6278c863e94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.101934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d8dbb-9e22-403a-b6f9-d6278c863e94 admin admin] PATH_INFO: `/v3/projects/1ae07dde35324ec2b54cf711fbadfec5/users/ffa0d522b31349b79725595818992c8a/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.114273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1022/2033] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:06 2022] DELETE /identity/v3/users/c634cb83513c4b4a8aff8a97cb1b7fc0 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:07.130784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8d8dbb-9e22-403a-b6f9-d6278c863e94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ae07dde35324ec2b54cf711fbadfec5, user_id=ffa0d522b31349b79725595818992c8a, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:07.133926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8d8dbb-9e22-403a-b6f9-d6278c863e94 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:07.151642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1012/2034] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:07 2022] PUT /identity/v3/projects/1ae07dde35324ec2b54cf711fbadfec5/users/ffa0d522b31349b79725595818992c8a/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:07.155736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:07.157352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:07.179925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:07.253960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26e2d9c-2053-412a-afab-17765dd74c67 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:07.346590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1501d0da-7a47-49ec-8fff-e2a5d1a262d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:07.347223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501d0da-7a47-49ec-8fff-e2a5d1a262d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.347529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501d0da-7a47-49ec-8fff-e2a5d1a262d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.347831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501d0da-7a47-49ec-8fff-e2a5d1a262d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.348783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1501d0da-7a47-49ec-8fff-e2a5d1a262d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:07.350602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1501d0da-7a47-49ec-8fff-e2a5d1a262d2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:07.374717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d8dbb-9e22-403a-b6f9-d6278c863e94 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:07.376529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1023/2035] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:07 2022] GET /identity/v3/roles => generated 1420 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:07.389325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d8dbb-9e22-403a-b6f9-d6278c863e94 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:07.396404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:07.397089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.397372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.397656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] PATH_INFO: `/v3/projects/1ae07dde35324ec2b54cf711fbadfec5/users/ffa0d522b31349b79725595818992c8a/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.400332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ae07dde35324ec2b54cf711fbadfec5, user_id=ffa0d522b31349b79725595818992c8a, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:07.403068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:07.416608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1013/2036] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:07 2022] PUT /identity/v3/projects/1ae07dde35324ec2b54cf711fbadfec5/users/ffa0d522b31349b79725595818992c8a/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:11:07.428538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a218b932-97dd-4204-a95c-d218dbefe09e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.428826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a218b932-97dd-4204-a95c-d218dbefe09e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.429102 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a218b932-97dd-4204-a95c-d218dbefe09e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.472983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a218b932-97dd-4204-a95c-d218dbefe09e None None] MFA Rules not processed for user `ffa0d522b31349b79725595818992c8a`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:07.474282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a218b932-97dd-4204-a95c-d218dbefe09e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:07.592015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1024/2037] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:07 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:07.605796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:07.609288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:07.676885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a218b932-97dd-4204-a95c-d218dbefe09e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:07.680993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a218b932-97dd-4204-a95c-d218dbefe09e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:07.741236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457160d8-f9fb-4015-800b-4a16ec293f94 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:07.746673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd4a683-123c-4ef0-830f-4e8656639320 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:07.747390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4a683-123c-4ef0-830f-4e8656639320 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.747798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4a683-123c-4ef0-830f-4e8656639320 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.748141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4a683-123c-4ef0-830f-4e8656639320 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.761224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd4a683-123c-4ef0-830f-4e8656639320 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:07.765507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd4a683-123c-4ef0-830f-4e8656639320 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:07.777126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1014/2038] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:07 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:07.783758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:07.806113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a218b932-97dd-4204-a95c-d218dbefe09e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:07.818050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:07.818745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.819021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.819289 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] PATH_INFO: `/v3/projects/c6f9a6da864d4670a84a0e712d526c81` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.820635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6f9a6da864d4670a84a0e712d526c81)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:07.822740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:07.865767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd4a683-123c-4ef0-830f-4e8656639320 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:07.877123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd4a683-123c-4ef0-830f-4e8656639320 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:07.889144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6f9a6da864d4670a84a0e712d526c81', 'request_id': 'req-ffeb510b-f927-4c3b-8c68-c6948c01b11f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.898664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6f9a6da864d4670a84a0e712d526c81', 'request_id': 'req-ffeb510b-f927-4c3b-8c68-c6948c01b11f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.906704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6f9a6da864d4670a84a0e712d526c81', 'request_id': 'req-ffeb510b-f927-4c3b-8c68-c6948c01b11f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.919526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] The token cache is being invalidate because project c6f9a6da864d4670a84a0e712d526c81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:07.919938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.921264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ffeb510b-f927-4c3b-8c68-c6948c01b11f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:07.923253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1025/2039] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:07 2022] DELETE /identity/v3/projects/c6f9a6da864d4670a84a0e712d526c81 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:07.927921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd98bdd-dfa8-483a-ad1d-4fc8e8244978 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.928291 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd98bdd-dfa8-483a-ad1d-4fc8e8244978 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.928626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd98bdd-dfa8-483a-ad1d-4fc8e8244978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.950848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2cd98bdd-dfa8-483a-ad1d-4fc8e8244978 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:07.952686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2cd98bdd-dfa8-483a-ad1d-4fc8e8244978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:07.968822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:07.969708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:07.969989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:07.970267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:07.974415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:08.062568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1026/2040] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:08.064694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e506a88-22e3-41fb-b36e-3a33e5f42423 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.067428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e506a88-22e3-41fb-b36e-3a33e5f42423 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.067428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e506a88-22e3-41fb-b36e-3a33e5f42423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.089516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9e506a88-22e3-41fb-b36e-3a33e5f42423 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:08.090784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9e506a88-22e3-41fb-b36e-3a33e5f42423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:08.103771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1027/2041] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:08.105637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e506a88-22e3-41fb-b36e-3a33e5f42423 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:08.115864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e506a88-22e3-41fb-b36e-3a33e5f42423 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:08.121597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:08.121867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.122155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.122429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.123094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:08.123399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.125553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:08.131069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.132687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:08.143731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1028/2042] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:11:08.147057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:08.152305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1015/2043] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:07 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:08.172993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1463995-2fd2-4002-9935-3e4d2344cb01 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:08.188721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3e08c4-8674-43d5-9c3e-8d4d71ec3ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:08.189529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3e08c4-8674-43d5-9c3e-8d4d71ec3ae1 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.189961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3e08c4-8674-43d5-9c3e-8d4d71ec3ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.190665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3e08c4-8674-43d5-9c3e-8d4d71ec3ae1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.191762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3e08c4-8674-43d5-9c3e-8d4d71ec3ae1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.194847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3e08c4-8674-43d5-9c3e-8d4d71ec3ae1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:08.288398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1029/2044] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:08 2022] POST /identity/v3/projects => generated 373 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:08.292861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:08.308930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9489fc61-1c7d-4065-ad74-ae32831833eb service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:08.337076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:08.416356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bcb3c5-f2ed-4eac-9087-3c67a168cc8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:08.416995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bcb3c5-f2ed-4eac-9087-3c67a168cc8c admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.417272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bcb3c5-f2ed-4eac-9087-3c67a168cc8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.417542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bcb3c5-f2ed-4eac-9087-3c67a168cc8c admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.420068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bcb3c5-f2ed-4eac-9087-3c67a168cc8c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.423742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bcb3c5-f2ed-4eac-9087-3c67a168cc8c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:08.459604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1016/2045] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:08 2022] POST /identity/v3/users => generated 287 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:08.463621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3e08c4-8674-43d5-9c3e-8d4d71ec3ae1 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:08.478082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3e08c4-8674-43d5-9c3e-8d4d71ec3ae1 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:08.484638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f387bfa9-0c6b-4c7d-ad86-327194b5f7d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:08.485406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f387bfa9-0c6b-4c7d-ad86-327194b5f7d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.485755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f387bfa9-0c6b-4c7d-ad86-327194b5f7d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.486731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f387bfa9-0c6b-4c7d-ad86-327194b5f7d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.487480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f387bfa9-0c6b-4c7d-ad86-327194b5f7d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.489405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f387bfa9-0c6b-4c7d-ad86-327194b5f7d0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:08.504153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1030/2046] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:08 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:08.505409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bcb3c5-f2ed-4eac-9087-3c67a168cc8c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:08.516588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bcb3c5-f2ed-4eac-9087-3c67a168cc8c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:08.521358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebca55a-6f53-4da7-8c70-ff8123ff3a83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:08.522112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebca55a-6f53-4da7-8c70-ff8123ff3a83 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.522397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebca55a-6f53-4da7-8c70-ff8123ff3a83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.522671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebca55a-6f53-4da7-8c70-ff8123ff3a83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.523327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebca55a-6f53-4da7-8c70-ff8123ff3a83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.524989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebca55a-6f53-4da7-8c70-ff8123ff3a83 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:08.536244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1017/2047] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:08.540547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f387bfa9-0c6b-4c7d-ad86-327194b5f7d0 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:08.551831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f387bfa9-0c6b-4c7d-ad86-327194b5f7d0 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:08.556609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:08.557361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.557834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.558440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] PATH_INFO: `/v3/projects/ef9c538579e742c9ae3cde1e6aaeb23e/users/864ee8a2a2104117ad23a81ac6861832/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.575719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef9c538579e742c9ae3cde1e6aaeb23e, user_id=864ee8a2a2104117ad23a81ac6861832, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.578827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:08.589296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:08.596265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1031/2048] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:08 2022] PUT /identity/v3/projects/ef9c538579e742c9ae3cde1e6aaeb23e/users/864ee8a2a2104117ad23a81ac6861832/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:08.608694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb49a10f-7985-4845-bd81-90374ee068a7 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.609562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb49a10f-7985-4845-bd81-90374ee068a7 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.609916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb49a10f-7985-4845-bd81-90374ee068a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.646822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-fb49a10f-7985-4845-bd81-90374ee068a7 None None] MFA Rules not processed for user `864ee8a2a2104117ad23a81ac6861832`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:08.648721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fb49a10f-7985-4845-bd81-90374ee068a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:08.753909 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1018/2049] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:08 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:08.765482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:08.767260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:08.840086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196e3e10-7437-453c-9903-ba1298294c52 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:08.844943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144d283c-b549-4caa-9552-bb8d026bfea6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:08.845389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d283c-b549-4caa-9552-bb8d026bfea6 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.845739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d283c-b549-4caa-9552-bb8d026bfea6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.846083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d283c-b549-4caa-9552-bb8d026bfea6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.856865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144d283c-b549-4caa-9552-bb8d026bfea6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.859030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144d283c-b549-4caa-9552-bb8d026bfea6 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:08.869302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1032/2050] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:08 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:08.892838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb49a10f-7985-4845-bd81-90374ee068a7 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:08.957669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb49a10f-7985-4845-bd81-90374ee068a7 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:08.962424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:08.963051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:08.963377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:08.963652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] PATH_INFO: `/v3/users/864ee8a2a2104117ad23a81ac6861832` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:08.964823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=864ee8a2a2104117ad23a81ac6861832)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:08.966733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:09.002455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:09.017970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '864ee8a2a2104117ad23a81ac6861832'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.025845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '864ee8a2a2104117ad23a81ac6861832'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.026457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '864ee8a2a2104117ad23a81ac6861832'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.027057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '864ee8a2a2104117ad23a81ac6861832'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.034531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '864ee8a2a2104117ad23a81ac6861832'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.052871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1019/2051] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:08 2022] DELETE /identity/v3/users/864ee8a2a2104117ad23a81ac6861832 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:09.057332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d671aa3-c7bf-4f70-8c88-544306ea5a14 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:09.057780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d671aa3-c7bf-4f70-8c88-544306ea5a14 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:09.058316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d671aa3-c7bf-4f70-8c88-544306ea5a14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:09.081208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1d671aa3-c7bf-4f70-8c88-544306ea5a14 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:09.082510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1d671aa3-c7bf-4f70-8c88-544306ea5a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:09.181419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1033/2052] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:09.192144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:09.194032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:09.262880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d795149c-49ed-4955-9c2b-b24599bd4b4d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:09.268116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c45ac58-920d-417a-af90-34f586450f72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:09.268732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45ac58-920d-417a-af90-34f586450f72 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:09.269012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45ac58-920d-417a-af90-34f586450f72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:09.269284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45ac58-920d-417a-af90-34f586450f72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:09.279540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c45ac58-920d-417a-af90-34f586450f72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:09.281814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c45ac58-920d-417a-af90-34f586450f72 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:09.296139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1020/2053] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:09.736497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d671aa3-c7bf-4f70-8c88-544306ea5a14 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:09.738049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1d671aa3-c7bf-4f70-8c88-544306ea5a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:09.753295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d671aa3-c7bf-4f70-8c88-544306ea5a14 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:09.758172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:09.758809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:09.759086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:09.759360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] PATH_INFO: `/v3/projects/ef9c538579e742c9ae3cde1e6aaeb23e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:09.760529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef9c538579e742c9ae3cde1e6aaeb23e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:09.762481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:09.827584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef9c538579e742c9ae3cde1e6aaeb23e', 'request_id': 'req-b8afbf8f-2863-4674-a587-2501d861db3e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.835087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef9c538579e742c9ae3cde1e6aaeb23e', 'request_id': 'req-b8afbf8f-2863-4674-a587-2501d861db3e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.840549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef9c538579e742c9ae3cde1e6aaeb23e', 'request_id': 'req-b8afbf8f-2863-4674-a587-2501d861db3e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.852671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] The token cache is being invalidate because project ef9c538579e742c9ae3cde1e6aaeb23e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:09.852957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.853603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b8afbf8f-2863-4674-a587-2501d861db3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:09.855098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1034/2054] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:09 2022] DELETE /identity/v3/projects/ef9c538579e742c9ae3cde1e6aaeb23e => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:09.860523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5729e74-b8b4-4339-8856-f8fd5bb6cdb9 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:09.861019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5729e74-b8b4-4339-8856-f8fd5bb6cdb9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:09.861112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5729e74-b8b4-4339-8856-f8fd5bb6cdb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:09.880525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e5729e74-b8b4-4339-8856-f8fd5bb6cdb9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:09.881781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e5729e74-b8b4-4339-8856-f8fd5bb6cdb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:09.943808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1021/2055] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:09.952936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaf16bc-c3b6-4df0-90c1-0086e583efd7 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:09.953235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaf16bc-c3b6-4df0-90c1-0086e583efd7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:09.953506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaf16bc-c3b6-4df0-90c1-0086e583efd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:09.973748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-fdaf16bc-c3b6-4df0-90c1-0086e583efd7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:09.974939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-fdaf16bc-c3b6-4df0-90c1-0086e583efd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:09.982660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1035/2056] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:09.986710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5729e74-b8b4-4339-8856-f8fd5bb6cdb9 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:09.996448 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5729e74-b8b4-4339-8856-f8fd5bb6cdb9 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.001612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.002250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.002530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.002801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.003459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:10.003796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.005453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.017605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1022/2057] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:11:10.021858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaf16bc-c3b6-4df0-90c1-0086e583efd7 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.031899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaf16bc-c3b6-4df0-90c1-0086e583efd7 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.036561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf73968-3bd9-4fce-843d-515df5b68ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.037148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf73968-3bd9-4fce-843d-515df5b68ce6 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.037426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf73968-3bd9-4fce-843d-515df5b68ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.037697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf73968-3bd9-4fce-843d-515df5b68ce6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.038331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf73968-3bd9-4fce-843d-515df5b68ce6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.040268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf73968-3bd9-4fce-843d-515df5b68ce6 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.055943 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:10.068610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1036/2058] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:10 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:10.072767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.083137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d85de68-4aa7-479d-8d48-f39fae1e58ec admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.115301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:10.140370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70323a1e-fc8a-4919-b59f-8787e2fb8a26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.140980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70323a1e-fc8a-4919-b59f-8787e2fb8a26 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.141264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70323a1e-fc8a-4919-b59f-8787e2fb8a26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.141532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70323a1e-fc8a-4919-b59f-8787e2fb8a26 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.142163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70323a1e-fc8a-4919-b59f-8787e2fb8a26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.144096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70323a1e-fc8a-4919-b59f-8787e2fb8a26 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.165353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1023/2059] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:10 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:10.169470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf73968-3bd9-4fce-843d-515df5b68ce6 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.180181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf73968-3bd9-4fce-843d-515df5b68ce6 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.185618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940228ff-2277-4506-9d4c-7dea36ea4983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.186225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940228ff-2277-4506-9d4c-7dea36ea4983 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.186628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940228ff-2277-4506-9d4c-7dea36ea4983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.187001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940228ff-2277-4506-9d4c-7dea36ea4983 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.187733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940228ff-2277-4506-9d4c-7dea36ea4983 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.189335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940228ff-2277-4506-9d4c-7dea36ea4983 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.199854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1037/2060] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:10.204669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70323a1e-fc8a-4919-b59f-8787e2fb8a26 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.214598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70323a1e-fc8a-4919-b59f-8787e2fb8a26 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.220097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1a79e1-0a95-4586-80e3-b9433cd3a746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.220773 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1a79e1-0a95-4586-80e3-b9433cd3a746 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.221056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1a79e1-0a95-4586-80e3-b9433cd3a746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.221331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1a79e1-0a95-4586-80e3-b9433cd3a746 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.222295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1a79e1-0a95-4586-80e3-b9433cd3a746 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.223902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1a79e1-0a95-4586-80e3-b9433cd3a746 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.234340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1024/2061] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:10.238293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940228ff-2277-4506-9d4c-7dea36ea4983 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.248471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940228ff-2277-4506-9d4c-7dea36ea4983 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.253118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431ab422-e574-4742-adcc-2ceecfcb10e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.253760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431ab422-e574-4742-adcc-2ceecfcb10e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.254038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431ab422-e574-4742-adcc-2ceecfcb10e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.254311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431ab422-e574-4742-adcc-2ceecfcb10e0 admin admin] PATH_INFO: `/v3/projects/99c29fe1ed3b4a209249ecc0615433bf/users/fa13fe5b7cd642b580503e00ee2fe6ad/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.269462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431ab422-e574-4742-adcc-2ceecfcb10e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=99c29fe1ed3b4a209249ecc0615433bf, user_id=fa13fe5b7cd642b580503e00ee2fe6ad, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.272175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431ab422-e574-4742-adcc-2ceecfcb10e0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.285319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1038/2062] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:10 2022] PUT /identity/v3/projects/99c29fe1ed3b4a209249ecc0615433bf/users/fa13fe5b7cd642b580503e00ee2fe6ad/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:10.295204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c002e78-dddd-4bda-9b9d-79efc55f40e4 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.295535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c002e78-dddd-4bda-9b9d-79efc55f40e4 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.295968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c002e78-dddd-4bda-9b9d-79efc55f40e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.329555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6c002e78-dddd-4bda-9b9d-79efc55f40e4 None None] MFA Rules not processed for user `fa13fe5b7cd642b580503e00ee2fe6ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:10.330827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6c002e78-dddd-4bda-9b9d-79efc55f40e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:10.409646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1025/2063] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:10 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:10.414155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431ab422-e574-4742-adcc-2ceecfcb10e0 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.486108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431ab422-e574-4742-adcc-2ceecfcb10e0 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.492725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa052c-fb84-4c6f-a388-8e1f255c1b34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.493403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa052c-fb84-4c6f-a388-8e1f255c1b34 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.493688 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa052c-fb84-4c6f-a388-8e1f255c1b34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.493961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa052c-fb84-4c6f-a388-8e1f255c1b34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.494702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0aa052c-fb84-4c6f-a388-8e1f255c1b34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.497081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0aa052c-fb84-4c6f-a388-8e1f255c1b34 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.524570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1039/2064] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:10 2022] POST /identity/v3/projects => generated 383 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:10.528644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c002e78-dddd-4bda-9b9d-79efc55f40e4 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.540075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c002e78-dddd-4bda-9b9d-79efc55f40e4 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.598360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f58882-3c0c-4117-b786-7e08eda1b32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.599159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f58882-3c0c-4117-b786-7e08eda1b32a admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.599504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f58882-3c0c-4117-b786-7e08eda1b32a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.599818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f58882-3c0c-4117-b786-7e08eda1b32a admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.600448 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f58882-3c0c-4117-b786-7e08eda1b32a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.602369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f58882-3c0c-4117-b786-7e08eda1b32a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.627562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1026/2065] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:10 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:10.628804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa052c-fb84-4c6f-a388-8e1f255c1b34 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.639005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa052c-fb84-4c6f-a388-8e1f255c1b34 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.643781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f924f2f6-4174-4d88-bb57-501d4c63f5bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.644401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f924f2f6-4174-4d88-bb57-501d4c63f5bd admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.644679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f924f2f6-4174-4d88-bb57-501d4c63f5bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.644947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f924f2f6-4174-4d88-bb57-501d4c63f5bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.645583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f924f2f6-4174-4d88-bb57-501d4c63f5bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.647180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f924f2f6-4174-4d88-bb57-501d4c63f5bd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.658506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1040/2066] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:10.662613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f58882-3c0c-4117-b786-7e08eda1b32a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.674377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f58882-3c0c-4117-b786-7e08eda1b32a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.678945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf17b69-cb04-4936-b93e-29d36075b9b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.679539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf17b69-cb04-4936-b93e-29d36075b9b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.680073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf17b69-cb04-4936-b93e-29d36075b9b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.680350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf17b69-cb04-4936-b93e-29d36075b9b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.681043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf17b69-cb04-4936-b93e-29d36075b9b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.682606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf17b69-cb04-4936-b93e-29d36075b9b1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.700394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1027/2067] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:10.700495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f924f2f6-4174-4d88-bb57-501d4c63f5bd admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.709063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f924f2f6-4174-4d88-bb57-501d4c63f5bd admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.715335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446c4c8-76a9-4a63-ad6f-ac357e58b2ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.716039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446c4c8-76a9-4a63-ad6f-ac357e58b2ee admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.716324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446c4c8-76a9-4a63-ad6f-ac357e58b2ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.716602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446c4c8-76a9-4a63-ad6f-ac357e58b2ee admin admin] PATH_INFO: `/v3/projects/a77d93616314423f94ee8979e57e8914/users/26de81a0c458470998663b9e2fa284d8/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.734326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3446c4c8-76a9-4a63-ad6f-ac357e58b2ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=a77d93616314423f94ee8979e57e8914, user_id=26de81a0c458470998663b9e2fa284d8, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.737117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3446c4c8-76a9-4a63-ad6f-ac357e58b2ee admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.750601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1041/2068] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:10 2022] PUT /identity/v3/projects/a77d93616314423f94ee8979e57e8914/users/26de81a0c458470998663b9e2fa284d8/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:10.760909 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fbf86e-fa4c-4eea-942e-95741bee33ef None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.761192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fbf86e-fa4c-4eea-942e-95741bee33ef None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.761471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fbf86e-fa4c-4eea-942e-95741bee33ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.790168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c0fbf86e-fa4c-4eea-942e-95741bee33ef None None] MFA Rules not processed for user `26de81a0c458470998663b9e2fa284d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:10.791392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c0fbf86e-fa4c-4eea-942e-95741bee33ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:10.803493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:10.862971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1028/2069] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:10 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:10.867517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446c4c8-76a9-4a63-ad6f-ac357e58b2ee admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:10.934909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446c4c8-76a9-4a63-ad6f-ac357e58b2ee admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:10.940468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478775bc-80f9-48d1-999e-0b6ee111aa76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:10.941075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478775bc-80f9-48d1-999e-0b6ee111aa76 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:10.941356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478775bc-80f9-48d1-999e-0b6ee111aa76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:10.941635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478775bc-80f9-48d1-999e-0b6ee111aa76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:10.942875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478775bc-80f9-48d1-999e-0b6ee111aa76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:10.945065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478775bc-80f9-48d1-999e-0b6ee111aa76 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:10.968495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:10.988808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1042/2070] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:10 2022] POST /identity/v3/projects => generated 383 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:10.992711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fbf86e-fa4c-4eea-942e-95741bee33ef None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:11.009987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fbf86e-fa4c-4eea-942e-95741bee33ef None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:11.078685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42536c3-2eaf-4787-a151-5510a9707f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:11.079364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42536c3-2eaf-4787-a151-5510a9707f66 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:11.079647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42536c3-2eaf-4787-a151-5510a9707f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:11.079970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42536c3-2eaf-4787-a151-5510a9707f66 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:11.080636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42536c3-2eaf-4787-a151-5510a9707f66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:11.082540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42536c3-2eaf-4787-a151-5510a9707f66 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:11.103596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1029/2071] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:10 2022] POST /identity/v3/users => generated 298 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:11.107498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478775bc-80f9-48d1-999e-0b6ee111aa76 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:11.118510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478775bc-80f9-48d1-999e-0b6ee111aa76 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:11.123081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1a733-4c1b-4625-8854-adac2b50e36b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:11.124286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1a733-4c1b-4625-8854-adac2b50e36b admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:11.124774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1a733-4c1b-4625-8854-adac2b50e36b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:11.125049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1a733-4c1b-4625-8854-adac2b50e36b admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:11.125724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b1a733-4c1b-4625-8854-adac2b50e36b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:11.127309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b1a733-4c1b-4625-8854-adac2b50e36b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:11.137715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1043/2072] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:11.142351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42536c3-2eaf-4787-a151-5510a9707f66 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:11.152405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42536c3-2eaf-4787-a151-5510a9707f66 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:11.157042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87d5b11-7c0b-44c8-b2bd-7df3e32cbee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:11.157687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87d5b11-7c0b-44c8-b2bd-7df3e32cbee2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:11.157967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87d5b11-7c0b-44c8-b2bd-7df3e32cbee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:11.158240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87d5b11-7c0b-44c8-b2bd-7df3e32cbee2 admin admin] PATH_INFO: `/v3/projects/4b549397a6344d628b75e7599b5c6810/users/e2bb8d5f1ccb4784943439a08e91e8c7/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:11.172794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87d5b11-7c0b-44c8-b2bd-7df3e32cbee2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b549397a6344d628b75e7599b5c6810, user_id=e2bb8d5f1ccb4784943439a08e91e8c7, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:11.175463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87d5b11-7c0b-44c8-b2bd-7df3e32cbee2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:11.189206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1030/2073] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:11 2022] PUT /identity/v3/projects/4b549397a6344d628b75e7599b5c6810/users/e2bb8d5f1ccb4784943439a08e91e8c7/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:11.193875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1a733-4c1b-4625-8854-adac2b50e36b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:11.206696 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1a733-4c1b-4625-8854-adac2b50e36b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:11.276914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab0f3f-5bcc-49f0-a049-bdff190ddf98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:11.278008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab0f3f-5bcc-49f0-a049-bdff190ddf98 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:11.278312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab0f3f-5bcc-49f0-a049-bdff190ddf98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:11.278595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab0f3f-5bcc-49f0-a049-bdff190ddf98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:11.279264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab0f3f-5bcc-49f0-a049-bdff190ddf98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:11.281732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab0f3f-5bcc-49f0-a049-bdff190ddf98 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:11.305287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1044/2074] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:11.306065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87d5b11-7c0b-44c8-b2bd-7df3e32cbee2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:11.321688 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87d5b11-7c0b-44c8-b2bd-7df3e32cbee2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:11.326707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:11.328122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:11.328401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:11.328680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] PATH_INFO: `/v3/projects/4b549397a6344d628b75e7599b5c6810/users/e2bb8d5f1ccb4784943439a08e91e8c7/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:11.330811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b549397a6344d628b75e7599b5c6810, user_id=e2bb8d5f1ccb4784943439a08e91e8c7, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:11.333571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:11.345615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1031/2075] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:11 2022] PUT /identity/v3/projects/4b549397a6344d628b75e7599b5c6810/users/e2bb8d5f1ccb4784943439a08e91e8c7/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:11.356341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668c9398-2fa5-42d5-93c2-31b0b1cc6992 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:11.356613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668c9398-2fa5-42d5-93c2-31b0b1cc6992 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:11.356883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668c9398-2fa5-42d5-93c2-31b0b1cc6992 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:11.397153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-668c9398-2fa5-42d5-93c2-31b0b1cc6992 None None] MFA Rules not processed for user `e2bb8d5f1ccb4784943439a08e91e8c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:11.398248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-668c9398-2fa5-42d5-93c2-31b0b1cc6992 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:11.519559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1045/2076] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:11 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:11.527862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:11.529593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:11.589153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e01b3ae-a349-4ae5-8e7f-8385a74c83e2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:11.592895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad3f339-a53b-48a1-bc23-81f8a9d54bf5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:11.593645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad3f339-a53b-48a1-bc23-81f8a9d54bf5 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:11.593923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad3f339-a53b-48a1-bc23-81f8a9d54bf5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:11.594194 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad3f339-a53b-48a1-bc23-81f8a9d54bf5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:11.613520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:11.654787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad3f339-a53b-48a1-bc23-81f8a9d54bf5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:11.656787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad3f339-a53b-48a1-bc23-81f8a9d54bf5 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:11.665972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1032/2077] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:11 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:14.291875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668c9398-2fa5-42d5-93c2-31b0b1cc6992 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:14.293253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-668c9398-2fa5-42d5-93c2-31b0b1cc6992 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:14.318478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668c9398-2fa5-42d5-93c2-31b0b1cc6992 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:14.321851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c341bc-ab33-4234-b643-5613b4af63cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:14.322462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c341bc-ab33-4234-b643-5613b4af63cd service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:14.322743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c341bc-ab33-4234-b643-5613b4af63cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:14.323019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c341bc-ab33-4234-b643-5613b4af63cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:14.383425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c341bc-ab33-4234-b643-5613b4af63cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:14.384898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c341bc-ab33-4234-b643-5613b4af63cd service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:14.394366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1046/2078] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:14 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:15.004694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad3f339-a53b-48a1-bc23-81f8a9d54bf5 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:15.014340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad3f339-a53b-48a1-bc23-81f8a9d54bf5 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:15.020960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb943f3-501a-43ef-9ca5-5a456b63f7f1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:15.021636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb943f3-501a-43ef-9ca5-5a456b63f7f1 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:15.021917 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb943f3-501a-43ef-9ca5-5a456b63f7f1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:15.022182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb943f3-501a-43ef-9ca5-5a456b63f7f1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:15.033051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb943f3-501a-43ef-9ca5-5a456b63f7f1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:15.034598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb943f3-501a-43ef-9ca5-5a456b63f7f1 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:15.047246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1033/2079] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:15 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:20.131125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c341bc-ab33-4234-b643-5613b4af63cd service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:20.136459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:20.197187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c341bc-ab33-4234-b643-5613b4af63cd service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:20.202044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:20.203277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:20.203427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:20.203775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] PATH_INFO: `/v3/users/fa13fe5b7cd642b580503e00ee2fe6ad` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:20.205725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fa13fe5b7cd642b580503e00ee2fe6ad)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:20.208176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:20.259376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fa13fe5b7cd642b580503e00ee2fe6ad'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.271152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fa13fe5b7cd642b580503e00ee2fe6ad'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.271820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fa13fe5b7cd642b580503e00ee2fe6ad'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.272457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fa13fe5b7cd642b580503e00ee2fe6ad'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.282339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fa13fe5b7cd642b580503e00ee2fe6ad'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.300653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1047/2080] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:20 2022] DELETE /identity/v3/users/fa13fe5b7cd642b580503e00ee2fe6ad => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:20.307916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86230baf-70f7-4bab-bdc6-71ffa88d26b2 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:20.308687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86230baf-70f7-4bab-bdc6-71ffa88d26b2 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:20.308781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86230baf-70f7-4bab-bdc6-71ffa88d26b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:20.328588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-86230baf-70f7-4bab-bdc6-71ffa88d26b2 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:20.329721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-86230baf-70f7-4bab-bdc6-71ffa88d26b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:20.392000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1034/2081] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:20.402201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:20.403667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:20.466217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd787c02-46dc-49b9-9366-cff992357cf6 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:20.471068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dff98f-9ef9-4c5d-968e-3619c4ff0c0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:20.471674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dff98f-9ef9-4c5d-968e-3619c4ff0c0f service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:20.472016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dff98f-9ef9-4c5d-968e-3619c4ff0c0f service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:20.472259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dff98f-9ef9-4c5d-968e-3619c4ff0c0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:20.482542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dff98f-9ef9-4c5d-968e-3619c4ff0c0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:20.484175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dff98f-9ef9-4c5d-968e-3619c4ff0c0f service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:20.494984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1048/2082] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:20.843514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86230baf-70f7-4bab-bdc6-71ffa88d26b2 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:20.844715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-86230baf-70f7-4bab-bdc6-71ffa88d26b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:20.860379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86230baf-70f7-4bab-bdc6-71ffa88d26b2 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:20.865362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:20.866230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:20.866697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:20.867153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] PATH_INFO: `/v3/projects/99c29fe1ed3b4a209249ecc0615433bf` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:20.869514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=99c29fe1ed3b4a209249ecc0615433bf)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:20.872845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:20.932356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '99c29fe1ed3b4a209249ecc0615433bf', 'request_id': 'req-66c5ed76-092f-404b-a181-ad2e53187a59'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.943114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '99c29fe1ed3b4a209249ecc0615433bf', 'request_id': 'req-66c5ed76-092f-404b-a181-ad2e53187a59'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.951321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '99c29fe1ed3b4a209249ecc0615433bf', 'request_id': 'req-66c5ed76-092f-404b-a181-ad2e53187a59'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.968717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] The token cache is being invalidate because project 99c29fe1ed3b4a209249ecc0615433bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:20.968717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.968717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:20.970849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1035/2083] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:20 2022] DELETE /identity/v3/projects/99c29fe1ed3b4a209249ecc0615433bf => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:20.974026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dff98f-9ef9-4c5d-968e-3619c4ff0c0f service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:20.988000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dff98f-9ef9-4c5d-968e-3619c4ff0c0f service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:21.023550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:21.072329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:21.073021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:21.073301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:21.073578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] PATH_INFO: `/v3/users/26de81a0c458470998663b9e2fa284d8` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:21.074764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] RBAC: Authorizing `identity:delete_user(user_id=26de81a0c458470998663b9e2fa284d8)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:21.078641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:21.127893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26de81a0c458470998663b9e2fa284d8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.138723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26de81a0c458470998663b9e2fa284d8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.139382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26de81a0c458470998663b9e2fa284d8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.140708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26de81a0c458470998663b9e2fa284d8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.148205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26de81a0c458470998663b9e2fa284d8'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.165576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1049/2084] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:20 2022] DELETE /identity/v3/users/26de81a0c458470998663b9e2fa284d8 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:21.469803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:21.471259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:21.484621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c5ed76-092f-404b-a181-ad2e53187a59 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:21.513441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:21.543855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:21.544482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:21.544761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:21.545032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] PATH_INFO: `/v3/projects/a77d93616314423f94ee8979e57e8914` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:21.546258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a77d93616314423f94ee8979e57e8914)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:21.548203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:21.595462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a77d93616314423f94ee8979e57e8914', 'request_id': 'req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.603312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a77d93616314423f94ee8979e57e8914', 'request_id': 'req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.609358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a77d93616314423f94ee8979e57e8914', 'request_id': 'req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.620562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] The token cache is being invalidate because project a77d93616314423f94ee8979e57e8914 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:21.620853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.621493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.622950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1036/2085] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:21 2022] DELETE /identity/v3/projects/a77d93616314423f94ee8979e57e8914 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:21.627481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:21.628910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:21.642605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139102b6-c395-46f9-8688-7a2f4475550a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:21.704752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:21.705385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:21.705668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:21.705942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] PATH_INFO: `/v3/users/e2bb8d5f1ccb4784943439a08e91e8c7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:21.707066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2bb8d5f1ccb4784943439a08e91e8c7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:21.709029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:21.763589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2bb8d5f1ccb4784943439a08e91e8c7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.766518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:21.774445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2bb8d5f1ccb4784943439a08e91e8c7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.775742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2bb8d5f1ccb4784943439a08e91e8c7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.777013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2bb8d5f1ccb4784943439a08e91e8c7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.785850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a1897b16-cd43-4111-9cc0-7a305607954a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2bb8d5f1ccb4784943439a08e91e8c7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:21.819166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1050/2086] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:21 2022] DELETE /identity/v3/users/e2bb8d5f1ccb4784943439a08e91e8c7 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:22.156104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:22.158532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:22.172412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d4b3f7-01d7-4a93-bd72-ae52cf6f2648 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:22.237488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:22.238046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.238328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.238604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] PATH_INFO: `/v3/projects/4b549397a6344d628b75e7599b5c6810` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.239744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b549397a6344d628b75e7599b5c6810)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:22.241745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:22.288510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b549397a6344d628b75e7599b5c6810', 'request_id': 'req-17d00df3-2449-4a10-af63-a570cfba4d7e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:22.295046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b549397a6344d628b75e7599b5c6810', 'request_id': 'req-17d00df3-2449-4a10-af63-a570cfba4d7e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:22.300041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:22.301524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b549397a6344d628b75e7599b5c6810', 'request_id': 'req-17d00df3-2449-4a10-af63-a570cfba4d7e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:22.315602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] The token cache is being invalidate because project 4b549397a6344d628b75e7599b5c6810 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:22.315926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:22.316578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-17d00df3-2449-4a10-af63-a570cfba4d7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:22.318039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1037/2087] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:22 2022] DELETE /identity/v3/projects/4b549397a6344d628b75e7599b5c6810 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:22.322756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d9d98-57c9-4732-b2c7-1374567ea360 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.323195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d9d98-57c9-4732-b2c7-1374567ea360 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.323285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d9d98-57c9-4732-b2c7-1374567ea360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.343837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ce3d9d98-57c9-4732-b2c7-1374567ea360 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:22.345174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ce3d9d98-57c9-4732-b2c7-1374567ea360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:22.408046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1051/2088] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:22.415778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80e3a-936b-443b-b504-64a8dd27f89b None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.416058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80e3a-936b-443b-b504-64a8dd27f89b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.416337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80e3a-936b-443b-b504-64a8dd27f89b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.437366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-03e80e3a-936b-443b-b504-64a8dd27f89b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:22.439425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-03e80e3a-936b-443b-b504-64a8dd27f89b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:22.448474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1038/2089] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:22.452666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3d9d98-57c9-4732-b2c7-1374567ea360 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:22.462993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3d9d98-57c9-4732-b2c7-1374567ea360 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:22.467624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:22.471548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.471934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.472216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.472944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:22.473249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:22.475029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:22.487794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1052/2090] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 14:11:22.491898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e80e3a-936b-443b-b504-64a8dd27f89b None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:22.502594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e80e3a-936b-443b-b504-64a8dd27f89b None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:22.507877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e54c17-9e69-4f92-8cbc-a649a2ed9a92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:22.508468 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e54c17-9e69-4f92-8cbc-a649a2ed9a92 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.508742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e54c17-9e69-4f92-8cbc-a649a2ed9a92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.509010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e54c17-9e69-4f92-8cbc-a649a2ed9a92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.509645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e54c17-9e69-4f92-8cbc-a649a2ed9a92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:22.511525 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e54c17-9e69-4f92-8cbc-a649a2ed9a92 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:22.539058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1039/2091] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:22 2022] POST /identity/v3/projects => generated 371 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:22.543392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:22.553401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37bd01a-30bd-408e-afb9-a07d87e37996 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:22.611271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08069e6-e8e5-43d9-8489-3855eca883f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:22.611908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08069e6-e8e5-43d9-8489-3855eca883f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.612195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08069e6-e8e5-43d9-8489-3855eca883f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.612470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08069e6-e8e5-43d9-8489-3855eca883f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.613094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08069e6-e8e5-43d9-8489-3855eca883f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:22.615012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08069e6-e8e5-43d9-8489-3855eca883f0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:22.636374 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1053/2092] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:22 2022] POST /identity/v3/users => generated 286 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:22.640517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e54c17-9e69-4f92-8cbc-a649a2ed9a92 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:22.651247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e54c17-9e69-4f92-8cbc-a649a2ed9a92 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:22.655824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab8040d-562b-4c49-b146-23f6107ec6ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:22.656423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab8040d-562b-4c49-b146-23f6107ec6ac admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.656696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab8040d-562b-4c49-b146-23f6107ec6ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.656975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab8040d-562b-4c49-b146-23f6107ec6ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.657635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab8040d-562b-4c49-b146-23f6107ec6ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:22.659218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab8040d-562b-4c49-b146-23f6107ec6ac admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:22.668953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1040/2093] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:22.673730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08069e6-e8e5-43d9-8489-3855eca883f0 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:22.683859 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08069e6-e8e5-43d9-8489-3855eca883f0 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:22.689248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae65d10-45f7-4a92-91ac-41aecf0d5afe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:22.689832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae65d10-45f7-4a92-91ac-41aecf0d5afe admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.690109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae65d10-45f7-4a92-91ac-41aecf0d5afe admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.690376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae65d10-45f7-4a92-91ac-41aecf0d5afe admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.691010 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae65d10-45f7-4a92-91ac-41aecf0d5afe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:22.692642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae65d10-45f7-4a92-91ac-41aecf0d5afe admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:22.702958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1054/2094] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:22.707061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab8040d-562b-4c49-b146-23f6107ec6ac admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:22.717055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab8040d-562b-4c49-b146-23f6107ec6ac admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:22.721641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:22.722281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.722561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.722836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] PATH_INFO: `/v3/projects/30154a777af94fb289cb7680459d85c3/users/a2ebafce9e4b4da5b2cffcfe8b77a686/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.737775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=30154a777af94fb289cb7680459d85c3, user_id=a2ebafce9e4b4da5b2cffcfe8b77a686, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:22.740519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:22.753547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1041/2095] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:22 2022] PUT /identity/v3/projects/30154a777af94fb289cb7680459d85c3/users/a2ebafce9e4b4da5b2cffcfe8b77a686/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:22.764703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a6c251-2a9e-4e7a-9587-2a206874db51 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.764989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a6c251-2a9e-4e7a-9587-2a206874db51 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.765256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a6c251-2a9e-4e7a-9587-2a206874db51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.793349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e5a6c251-2a9e-4e7a-9587-2a206874db51 None None] MFA Rules not processed for user `a2ebafce9e4b4da5b2cffcfe8b77a686`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:22.794546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e5a6c251-2a9e-4e7a-9587-2a206874db51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:22.813729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:22.878646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1055/2096] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:22 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:22.890538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:22.892458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:22.951132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e83146-1283-4c84-a176-e6d998e0a3fc admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:22.955692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c70b7-adc0-4875-903a-72680fc62835 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:22.956329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c70b7-adc0-4875-903a-72680fc62835 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:22.956611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c70b7-adc0-4875-903a-72680fc62835 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:22.956879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c70b7-adc0-4875-903a-72680fc62835 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:22.965168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c70b7-adc0-4875-903a-72680fc62835 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:22.966583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c70b7-adc0-4875-903a-72680fc62835 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:22.976970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1042/2097] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:22 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:23.003279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a6c251-2a9e-4e7a-9587-2a206874db51 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:23.068182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a6c251-2a9e-4e7a-9587-2a206874db51 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:23.073123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:23.073820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.074168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.074509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] PATH_INFO: `/v3/users/a2ebafce9e4b4da5b2cffcfe8b77a686` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.075746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2ebafce9e4b4da5b2cffcfe8b77a686)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:23.077778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:23.134094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2ebafce9e4b4da5b2cffcfe8b77a686'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.142501 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2ebafce9e4b4da5b2cffcfe8b77a686'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.143288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2ebafce9e4b4da5b2cffcfe8b77a686'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.143995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2ebafce9e4b4da5b2cffcfe8b77a686'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.151975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2ebafce9e4b4da5b2cffcfe8b77a686'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.170692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1056/2098] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:23 2022] DELETE /identity/v3/users/a2ebafce9e4b4da5b2cffcfe8b77a686 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:23.176138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a01e9b-b9fa-4839-8594-9d98ac28dc77 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.176428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a01e9b-b9fa-4839-8594-9d98ac28dc77 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.176701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a01e9b-b9fa-4839-8594-9d98ac28dc77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.198699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e6a01e9b-b9fa-4839-8594-9d98ac28dc77 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:23.199973 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e6a01e9b-b9fa-4839-8594-9d98ac28dc77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:23.270636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1043/2099] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:23.282211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:23.284197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:23.362794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:23.387352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a764f5-8200-439d-afb7-ded7248dc6fc admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:23.394120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f926b4-cec5-4ad2-8801-77443965bce0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:23.394750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f926b4-cec5-4ad2-8801-77443965bce0 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.395012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f926b4-cec5-4ad2-8801-77443965bce0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.395301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f926b4-cec5-4ad2-8801-77443965bce0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.404759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f926b4-cec5-4ad2-8801-77443965bce0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:23.406181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f926b4-cec5-4ad2-8801-77443965bce0 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:23.416445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1057/2100] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:23.507660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a01e9b-b9fa-4839-8594-9d98ac28dc77 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:23.509097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e6a01e9b-b9fa-4839-8594-9d98ac28dc77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:23.522142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a01e9b-b9fa-4839-8594-9d98ac28dc77 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:23.526816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:23.527450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.527763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.528032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] PATH_INFO: `/v3/users/f6c928f94dfe400fa48e6d4e5bc9638c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.529144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6c928f94dfe400fa48e6d4e5bc9638c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:23.531090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:23.559140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:23.586310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6c928f94dfe400fa48e6d4e5bc9638c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.595989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6c928f94dfe400fa48e6d4e5bc9638c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.596630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6c928f94dfe400fa48e6d4e5bc9638c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.597407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6c928f94dfe400fa48e6d4e5bc9638c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.604893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6c928f94dfe400fa48e6d4e5bc9638c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.622545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1044/2101] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:23 2022] DELETE /identity/v3/users/f6c928f94dfe400fa48e6d4e5bc9638c => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:23.627364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d448e9-36e4-4fa7-afb4-c624e70e23e4 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.627809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d448e9-36e4-4fa7-afb4-c624e70e23e4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.628095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d448e9-36e4-4fa7-afb4-c624e70e23e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.648304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-33d448e9-36e4-4fa7-afb4-c624e70e23e4 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:23.649625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-33d448e9-36e4-4fa7-afb4-c624e70e23e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:23.695503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:23.697007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:23.727647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1058/2102] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:23.733913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d448e9-36e4-4fa7-afb4-c624e70e23e4 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:23.735828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-33d448e9-36e4-4fa7-afb4-c624e70e23e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:23.770643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa407f34-b0d8-48a5-9e4c-e6fe92f940a7 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:23.775129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:23.775833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.776114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.776381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] PATH_INFO: `/v3/projects/30154a777af94fb289cb7680459d85c3` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.777490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] RBAC: Authorizing `identity:delete_project(project_id=30154a777af94fb289cb7680459d85c3)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:23.779386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:23.793040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d448e9-36e4-4fa7-afb4-c624e70e23e4 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:23.796616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261c3534-9f50-48c3-8f6f-e6bbaaa17910 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:23.797259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261c3534-9f50-48c3-8f6f-e6bbaaa17910 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.797538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261c3534-9f50-48c3-8f6f-e6bbaaa17910 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.797811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261c3534-9f50-48c3-8f6f-e6bbaaa17910 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.806944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261c3534-9f50-48c3-8f6f-e6bbaaa17910 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:23.808890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261c3534-9f50-48c3-8f6f-e6bbaaa17910 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:23.819339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1059/2103] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:23.828492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30154a777af94fb289cb7680459d85c3', 'request_id': 'req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.836019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30154a777af94fb289cb7680459d85c3', 'request_id': 'req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.841786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30154a777af94fb289cb7680459d85c3', 'request_id': 'req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.854098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] The token cache is being invalidate because project 30154a777af94fb289cb7680459d85c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:23.854390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.855074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4bed911-5b38-47a4-94a9-c43c3ba2c57a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:23.856828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1045/2104] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:23 2022] DELETE /identity/v3/projects/30154a777af94fb289cb7680459d85c3 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:11:23.862202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0654bb8-f3f5-4d1a-94d0-1e71ca194c34 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.862488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0654bb8-f3f5-4d1a-94d0-1e71ca194c34 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.862767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0654bb8-f3f5-4d1a-94d0-1e71ca194c34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.884344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d0654bb8-f3f5-4d1a-94d0-1e71ca194c34 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:23.885609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d0654bb8-f3f5-4d1a-94d0-1e71ca194c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:23.959878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1060/2105] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:23.967649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f560a2bb-9701-41a0-8d48-04cce6381aca None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:23.967969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f560a2bb-9701-41a0-8d48-04cce6381aca None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:23.968251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f560a2bb-9701-41a0-8d48-04cce6381aca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:23.990948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f560a2bb-9701-41a0-8d48-04cce6381aca None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:23.992248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f560a2bb-9701-41a0-8d48-04cce6381aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:24.001645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1046/2106] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:24.006331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0654bb8-f3f5-4d1a-94d0-1e71ca194c34 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:24.017380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0654bb8-f3f5-4d1a-94d0-1e71ca194c34 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:24.022243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:24.022924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.023208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.023485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.024251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:24.024557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:24.026164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:24.038416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1061/2107] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:11:24.042709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f560a2bb-9701-41a0-8d48-04cce6381aca None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:24.052251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f560a2bb-9701-41a0-8d48-04cce6381aca None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:24.056869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:24.057464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.057736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.058003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.058899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:24.060832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:24.091725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1047/2108] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:24 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:24.093801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:24.102897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:24.104561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:24.104910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ae0b0c-04b0-4983-932a-b628549cd8db admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:24.111754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:24.114921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:24.177882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e9dcde-4095-4fd3-b2b2-c38030ba50ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:24.178714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e9dcde-4095-4fd3-b2b2-c38030ba50ea admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.178976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e9dcde-4095-4fd3-b2b2-c38030ba50ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.179239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e9dcde-4095-4fd3-b2b2-c38030ba50ea admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.179895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e9dcde-4095-4fd3-b2b2-c38030ba50ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:24.181818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e9dcde-4095-4fd3-b2b2-c38030ba50ea admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:24.185592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0933f1c0-34e3-4b61-91ff-224e9210337b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:24.190944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:24.191589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.191888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.192163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] PATH_INFO: `/v3/projects/57a2d639abb946318d3ff8fd4924f977` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.193297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=57a2d639abb946318d3ff8fd4924f977)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:24.195201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:24.207994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1062/2109] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:24 2022] POST /identity/v3/users => generated 292 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:24.212020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e9dcde-4095-4fd3-b2b2-c38030ba50ea admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:24.224931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e9dcde-4095-4fd3-b2b2-c38030ba50ea admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:24.230651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a3f828-da20-4f6c-bd03-58d4412ac8cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:24.231244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a3f828-da20-4f6c-bd03-58d4412ac8cc admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.231543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a3f828-da20-4f6c-bd03-58d4412ac8cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.231847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a3f828-da20-4f6c-bd03-58d4412ac8cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.232492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a3f828-da20-4f6c-bd03-58d4412ac8cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:24.234094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a3f828-da20-4f6c-bd03-58d4412ac8cc admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:24.247293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1063/2110] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:24 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:11:24.254077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a3f828-da20-4f6c-bd03-58d4412ac8cc admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:24.254259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57a2d639abb946318d3ff8fd4924f977', 'request_id': 'req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:24.265627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57a2d639abb946318d3ff8fd4924f977', 'request_id': 'req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:24.267057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a3f828-da20-4f6c-bd03-58d4412ac8cc admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:24.274389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57a2d639abb946318d3ff8fd4924f977', 'request_id': 'req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:24.289444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] The token cache is being invalidate because project 57a2d639abb946318d3ff8fd4924f977 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:24.289742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:24.290420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:24.291981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1048/2111] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:24 2022] DELETE /identity/v3/projects/57a2d639abb946318d3ff8fd4924f977 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:11:24.353099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf7c2ab-2cce-43a8-8b21-c98d44e9913c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:24.353720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf7c2ab-2cce-43a8-8b21-c98d44e9913c admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.353995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf7c2ab-2cce-43a8-8b21-c98d44e9913c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.354264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf7c2ab-2cce-43a8-8b21-c98d44e9913c admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.354914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf7c2ab-2cce-43a8-8b21-c98d44e9913c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:24.356570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf7c2ab-2cce-43a8-8b21-c98d44e9913c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:24.369682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1064/2112] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:24 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:24.373916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:24.376384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:24.398340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5bc94-38e5-4ef2-b17d-e7153a2d31ec admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:24.413883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:24.414589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.414873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.415159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] PATH_INFO: `/v3/projects/1e90b31934484572bf7c7bdcbc0fdf2d/users/0ad0877d583d4550b9ab14ee7f4394f0/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.436957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e90b31934484572bf7c7bdcbc0fdf2d, user_id=0ad0877d583d4550b9ab14ee7f4394f0, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:24.442326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:24.467021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1049/2113] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:24 2022] PUT /identity/v3/projects/1e90b31934484572bf7c7bdcbc0fdf2d/users/0ad0877d583d4550b9ab14ee7f4394f0/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:24.469379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f4eb20-1254-432d-be8e-37ce4d9ddb50 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.469668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f4eb20-1254-432d-be8e-37ce4d9ddb50 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.469941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f4eb20-1254-432d-be8e-37ce4d9ddb50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.503794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-60f4eb20-1254-432d-be8e-37ce4d9ddb50 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:24.505249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-60f4eb20-1254-432d-be8e-37ce4d9ddb50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:24.583218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1065/2114] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:24.595336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:24.597158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:24.664042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfe677e-06c9-440d-8588-af157a913063 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:24.667823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:24.668434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:24.668716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:24.668999 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:24.678860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:24.683868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:24.690299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1050/2115] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:25.327337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12b1ae-9cbe-4ab7-b863-b062fdf734b8 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:25.327692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12b1ae-9cbe-4ab7-b863-b062fdf734b8 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:25.327984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12b1ae-9cbe-4ab7-b863-b062fdf734b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:25.361554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-fc12b1ae-9cbe-4ab7-b863-b062fdf734b8 None None] MFA Rules not processed for user `0ad0877d583d4550b9ab14ee7f4394f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:25.362770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-fc12b1ae-9cbe-4ab7-b863-b062fdf734b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:25.380283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:25.441083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1066/2116] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:25.451821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:25.453856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:25.467824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:25.513975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493de670-4bd9-47ef-8f24-4ecde28b2674 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:25.518173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:25.518730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:25.519005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:25.519273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:25.527624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:25.529054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:25.537650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1051/2117] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:25 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:33.246675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f969dec-de05-4ff3-b75a-a4480cc31abd None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:33.247333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f969dec-de05-4ff3-b75a-a4480cc31abd None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:33.247333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f969dec-de05-4ff3-b75a-a4480cc31abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:33.272326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2f969dec-de05-4ff3-b75a-a4480cc31abd None None] MFA Rules not processed for user `0ad0877d583d4550b9ab14ee7f4394f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:33.273674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2f969dec-de05-4ff3-b75a-a4480cc31abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:33.280781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1067/2118] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:33 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:34.608177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:34.609743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:34.623091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e87037f-9508-4ee2-885a-6193e01e4c1f service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:34.628108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:34.628720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:34.629003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:34.629278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] PATH_INFO: `/v3/users/7b9670eefd0c49d38a3ff91505ff7bc7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:34.630388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b9670eefd0c49d38a3ff91505ff7bc7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:34.632341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:34.676579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b9670eefd0c49d38a3ff91505ff7bc7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:34.683948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b9670eefd0c49d38a3ff91505ff7bc7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:34.684574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b9670eefd0c49d38a3ff91505ff7bc7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:34.685206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b9670eefd0c49d38a3ff91505ff7bc7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:34.693149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b9670eefd0c49d38a3ff91505ff7bc7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:34.710309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1052/2119] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:34 2022] DELETE /identity/v3/users/7b9670eefd0c49d38a3ff91505ff7bc7 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:34.714644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534d8da-ae00-49b1-8e9a-f0afdd5a93e4 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:34.715138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534d8da-ae00-49b1-8e9a-f0afdd5a93e4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:34.715240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534d8da-ae00-49b1-8e9a-f0afdd5a93e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:34.733818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c534d8da-ae00-49b1-8e9a-f0afdd5a93e4 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:34.735588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c534d8da-ae00-49b1-8e9a-f0afdd5a93e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:34.793575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1068/2120] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:34.803952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:34.805425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:34.861899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90affc53-087d-423f-a3ec-7dce09a29269 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:34.865841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80470dc-f210-4852-98b6-03070064eff4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:34.866443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80470dc-f210-4852-98b6-03070064eff4 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:34.866720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80470dc-f210-4852-98b6-03070064eff4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:34.866991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80470dc-f210-4852-98b6-03070064eff4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:34.876406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80470dc-f210-4852-98b6-03070064eff4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:34.877802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80470dc-f210-4852-98b6-03070064eff4 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:34.886886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1053/2121] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:11:35.162449 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c534d8da-ae00-49b1-8e9a-f0afdd5a93e4 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:35.164322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c534d8da-ae00-49b1-8e9a-f0afdd5a93e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:35.201491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c534d8da-ae00-49b1-8e9a-f0afdd5a93e4 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:35.218088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:35.219373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:35.219754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:35.220125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] PATH_INFO: `/v3/projects/8e6d672c90fc4025a094d90f76c39e2c` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:35.221377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e6d672c90fc4025a094d90f76c39e2c)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:35.225258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:35.304185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e6d672c90fc4025a094d90f76c39e2c', 'request_id': 'req-45b76c90-cb20-412f-9104-b796dee2ce35'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.320616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e6d672c90fc4025a094d90f76c39e2c', 'request_id': 'req-45b76c90-cb20-412f-9104-b796dee2ce35'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.334411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e6d672c90fc4025a094d90f76c39e2c', 'request_id': 'req-45b76c90-cb20-412f-9104-b796dee2ce35'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.347024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] The token cache is being invalidate because project 8e6d672c90fc4025a094d90f76c39e2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:35.347318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.348039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.349639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1069/2122] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:35 2022] DELETE /identity/v3/projects/8e6d672c90fc4025a094d90f76c39e2c => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:35.353798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80470dc-f210-4852-98b6-03070064eff4 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:35.364074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80470dc-f210-4852-98b6-03070064eff4 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:35.392831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:35.425210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:35.425840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:35.426132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:35.426401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] PATH_INFO: `/v3/users/ffa0d522b31349b79725595818992c8a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:35.427520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffa0d522b31349b79725595818992c8a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:35.429891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:35.477254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffa0d522b31349b79725595818992c8a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.485190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffa0d522b31349b79725595818992c8a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.485786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffa0d522b31349b79725595818992c8a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.486376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffa0d522b31349b79725595818992c8a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.493643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9d7956a1-9090-4f60-b438-760d17c07367 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffa0d522b31349b79725595818992c8a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.511119 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1054/2123] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:35 2022] DELETE /identity/v3/users/ffa0d522b31349b79725595818992c8a => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:35.802913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:35.804529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:35.822562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b76c90-cb20-412f-9104-b796dee2ce35 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:35.861922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:35.887250 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:35.887946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:35.888230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:35.888502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] PATH_INFO: `/v3/projects/1ae07dde35324ec2b54cf711fbadfec5` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:35.889644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ae07dde35324ec2b54cf711fbadfec5)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:35.891586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:35.940418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ae07dde35324ec2b54cf711fbadfec5', 'request_id': 'req-be7751db-a987-45a2-9dbc-7a480d167d69'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.947491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ae07dde35324ec2b54cf711fbadfec5', 'request_id': 'req-be7751db-a987-45a2-9dbc-7a480d167d69'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.953050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ae07dde35324ec2b54cf711fbadfec5', 'request_id': 'req-be7751db-a987-45a2-9dbc-7a480d167d69'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.964214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] The token cache is being invalidate because project 1ae07dde35324ec2b54cf711fbadfec5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:35.964499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.965144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-be7751db-a987-45a2-9dbc-7a480d167d69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:35.966604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1070/2124] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:35 2022] DELETE /identity/v3/projects/1ae07dde35324ec2b54cf711fbadfec5 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:35.972007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5419ac-06ba-42f1-80f7-a99bed22a4d0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:35.972500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5419ac-06ba-42f1-80f7-a99bed22a4d0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:35.972604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5419ac-06ba-42f1-80f7-a99bed22a4d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:35.992288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-eb5419ac-06ba-42f1-80f7-a99bed22a4d0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:35.993474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-eb5419ac-06ba-42f1-80f7-a99bed22a4d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:36.051936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1055/2125] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:36.060390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98246b00-da82-4bbf-b6b5-37bac6723e23 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.060676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98246b00-da82-4bbf-b6b5-37bac6723e23 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.060957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98246b00-da82-4bbf-b6b5-37bac6723e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.080744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-98246b00-da82-4bbf-b6b5-37bac6723e23 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:36.082002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-98246b00-da82-4bbf-b6b5-37bac6723e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:36.090021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1071/2126] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:36.095104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5419ac-06ba-42f1-80f7-a99bed22a4d0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.105134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5419ac-06ba-42f1-80f7-a99bed22a4d0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.110619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.111231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.111582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.111959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.112681 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:36.113072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.114733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.126449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1056/2127] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:11:36.130783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98246b00-da82-4bbf-b6b5-37bac6723e23 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.141014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98246b00-da82-4bbf-b6b5-37bac6723e23 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.145186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4258ede0-cf73-4192-a8eb-a95f93c4c7f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.145792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4258ede0-cf73-4192-a8eb-a95f93c4c7f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.146074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4258ede0-cf73-4192-a8eb-a95f93c4c7f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.146349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4258ede0-cf73-4192-a8eb-a95f93c4c7f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.147005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4258ede0-cf73-4192-a8eb-a95f93c4c7f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.148945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4258ede0-cf73-4192-a8eb-a95f93c4c7f3 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.176411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1072/2128] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:36 2022] POST /identity/v3/projects => generated 367 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:36.180597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.190906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d2e863-14a5-4be9-afdf-e97b6782259b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.226342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:36.252882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6c9c8-fd46-4e85-b209-de49e58883f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.253503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6c9c8-fd46-4e85-b209-de49e58883f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.253786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6c9c8-fd46-4e85-b209-de49e58883f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.254058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6c9c8-fd46-4e85-b209-de49e58883f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.254699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f6c9c8-fd46-4e85-b209-de49e58883f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.256674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f6c9c8-fd46-4e85-b209-de49e58883f1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.283513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1057/2129] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:36 2022] POST /identity/v3/users => generated 284 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:36.287660 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4258ede0-cf73-4192-a8eb-a95f93c4c7f3 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.311791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4258ede0-cf73-4192-a8eb-a95f93c4c7f3 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.325070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0554ce15-6b88-4e1e-b958-4f186935ff18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.325786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0554ce15-6b88-4e1e-b958-4f186935ff18 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.326244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0554ce15-6b88-4e1e-b958-4f186935ff18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.326585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0554ce15-6b88-4e1e-b958-4f186935ff18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.327364 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0554ce15-6b88-4e1e-b958-4f186935ff18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.329584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0554ce15-6b88-4e1e-b958-4f186935ff18 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.341287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1073/2130] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:11:36.346517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6c9c8-fd46-4e85-b209-de49e58883f1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.357686 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6c9c8-fd46-4e85-b209-de49e58883f1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.363171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135e2ba3-5369-4873-b3e5-1378ab68b20b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.363755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135e2ba3-5369-4873-b3e5-1378ab68b20b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.364045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135e2ba3-5369-4873-b3e5-1378ab68b20b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.364320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135e2ba3-5369-4873-b3e5-1378ab68b20b admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.364987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135e2ba3-5369-4873-b3e5-1378ab68b20b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.366732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135e2ba3-5369-4873-b3e5-1378ab68b20b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.381744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1058/2131] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:36.386113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0554ce15-6b88-4e1e-b958-4f186935ff18 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.398855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0554ce15-6b88-4e1e-b958-4f186935ff18 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.424334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794d2530-6580-46e0-b070-4068bcf5afc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.425051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794d2530-6580-46e0-b070-4068bcf5afc6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.425340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794d2530-6580-46e0-b070-4068bcf5afc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.425686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794d2530-6580-46e0-b070-4068bcf5afc6 admin admin] PATH_INFO: `/v3/projects/e338eb5f015c4b83904f9783faa1b548/users/fca684dce8ea4fc48b13f69595289024/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.464973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794d2530-6580-46e0-b070-4068bcf5afc6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e338eb5f015c4b83904f9783faa1b548, user_id=fca684dce8ea4fc48b13f69595289024, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.468335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794d2530-6580-46e0-b070-4068bcf5afc6 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.486385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1074/2132] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:36 2022] PUT /identity/v3/projects/e338eb5f015c4b83904f9783faa1b548/users/fca684dce8ea4fc48b13f69595289024/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:36.497920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100da440-2626-43b3-9f90-b3bf92834bfb None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.498236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100da440-2626-43b3-9f90-b3bf92834bfb None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.498483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100da440-2626-43b3-9f90-b3bf92834bfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.529386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-100da440-2626-43b3-9f90-b3bf92834bfb None None] MFA Rules not processed for user `fca684dce8ea4fc48b13f69595289024`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:36.530619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-100da440-2626-43b3-9f90-b3bf92834bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:36.625296 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1059/2133] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:36 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:36.629629 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794d2530-6580-46e0-b070-4068bcf5afc6 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.696223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794d2530-6580-46e0-b070-4068bcf5afc6 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.700797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f04337-a210-4c05-9348-5dc59b66242c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.701399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f04337-a210-4c05-9348-5dc59b66242c admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.701677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f04337-a210-4c05-9348-5dc59b66242c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.701956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f04337-a210-4c05-9348-5dc59b66242c admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.702595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f04337-a210-4c05-9348-5dc59b66242c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.705721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f04337-a210-4c05-9348-5dc59b66242c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.711746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:36.735410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1075/2134] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:36 2022] POST /identity/v3/projects => generated 367 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:36.740022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100da440-2626-43b3-9f90-b3bf92834bfb None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.752540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100da440-2626-43b3-9f90-b3bf92834bfb None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.812934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0fc4d7-1d01-4ef2-8543-d6a2aba33bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.813812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0fc4d7-1d01-4ef2-8543-d6a2aba33bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.814097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0fc4d7-1d01-4ef2-8543-d6a2aba33bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.814372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0fc4d7-1d01-4ef2-8543-d6a2aba33bfc admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.815050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0fc4d7-1d01-4ef2-8543-d6a2aba33bfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.817174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0fc4d7-1d01-4ef2-8543-d6a2aba33bfc admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.842797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1060/2135] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:36 2022] POST /identity/v3/users => generated 290 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:36.844670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f04337-a210-4c05-9348-5dc59b66242c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.856510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f04337-a210-4c05-9348-5dc59b66242c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.862169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b1afb-b2ee-4ebc-b3c3-3009bb0bfe4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.862776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b1afb-b2ee-4ebc-b3c3-3009bb0bfe4b admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.863060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b1afb-b2ee-4ebc-b3c3-3009bb0bfe4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.863335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b1afb-b2ee-4ebc-b3c3-3009bb0bfe4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.864020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672b1afb-b2ee-4ebc-b3c3-3009bb0bfe4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.865646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672b1afb-b2ee-4ebc-b3c3-3009bb0bfe4b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.875974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1076/2136] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:11:36.880307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0fc4d7-1d01-4ef2-8543-d6a2aba33bfc admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.890409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0fc4d7-1d01-4ef2-8543-d6a2aba33bfc admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:36.895137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b807e20-91e9-4c00-a591-f0b289adfe5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:36.895916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b807e20-91e9-4c00-a591-f0b289adfe5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:36.896267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b807e20-91e9-4c00-a591-f0b289adfe5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:36.896549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b807e20-91e9-4c00-a591-f0b289adfe5a admin admin] PATH_INFO: `/v3/projects/ae341e18bba840f3839da2e926f41b17/users/df7f7e9f4e954a198bcfcc52c2a2e7e7/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:36.913234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b807e20-91e9-4c00-a591-f0b289adfe5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae341e18bba840f3839da2e926f41b17, user_id=df7f7e9f4e954a198bcfcc52c2a2e7e7, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:36.916263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b807e20-91e9-4c00-a591-f0b289adfe5a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:36.932437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1061/2137] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:36 2022] PUT /identity/v3/projects/ae341e18bba840f3839da2e926f41b17/users/df7f7e9f4e954a198bcfcc52c2a2e7e7/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:36.936725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b1afb-b2ee-4ebc-b3c3-3009bb0bfe4b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:36.947386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b1afb-b2ee-4ebc-b3c3-3009bb0bfe4b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:37.007191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8099b0a6-7a6f-4984-bc39-680bbd3f7afd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:37.007995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8099b0a6-7a6f-4984-bc39-680bbd3f7afd admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:37.008374 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8099b0a6-7a6f-4984-bc39-680bbd3f7afd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:37.008747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8099b0a6-7a6f-4984-bc39-680bbd3f7afd admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:37.009490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8099b0a6-7a6f-4984-bc39-680bbd3f7afd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:37.011199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8099b0a6-7a6f-4984-bc39-680bbd3f7afd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:37.024398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1077/2138] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:11:37.025549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b807e20-91e9-4c00-a591-f0b289adfe5a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:37.036307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b807e20-91e9-4c00-a591-f0b289adfe5a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:37.040941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:37.041631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:37.042010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:37.042290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] PATH_INFO: `/v3/projects/ae341e18bba840f3839da2e926f41b17/users/df7f7e9f4e954a198bcfcc52c2a2e7e7/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:37.044211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae341e18bba840f3839da2e926f41b17, user_id=df7f7e9f4e954a198bcfcc52c2a2e7e7, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:37.047637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:37.054070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:37.065330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1062/2139] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:37 2022] PUT /identity/v3/projects/ae341e18bba840f3839da2e926f41b17/users/df7f7e9f4e954a198bcfcc52c2a2e7e7/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:37.075257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58a0ae-19e0-4553-86dd-5262341bb7c9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:37.075603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58a0ae-19e0-4553-86dd-5262341bb7c9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:37.075923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58a0ae-19e0-4553-86dd-5262341bb7c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:37.106413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-cc58a0ae-19e0-4553-86dd-5262341bb7c9 None None] MFA Rules not processed for user `df7f7e9f4e954a198bcfcc52c2a2e7e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:37.107621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-cc58a0ae-19e0-4553-86dd-5262341bb7c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:37.183076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1078/2140] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:37 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:37.194865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:37.196697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:37.270588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f4826d-8d77-4327-9a09-48da321a0645 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:37.275159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e32033-fb17-4f80-b93d-feb3a2930a09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:37.275797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e32033-fb17-4f80-b93d-feb3a2930a09 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:37.276075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e32033-fb17-4f80-b93d-feb3a2930a09 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:37.276346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e32033-fb17-4f80-b93d-feb3a2930a09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:37.287724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e32033-fb17-4f80-b93d-feb3a2930a09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:37.289222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e32033-fb17-4f80-b93d-feb3a2930a09 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:37.304837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1063/2141] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:37 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:37.485677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58a0ae-19e0-4553-86dd-5262341bb7c9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:37.488326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-cc58a0ae-19e0-4553-86dd-5262341bb7c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:37.556830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:37.561846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58a0ae-19e0-4553-86dd-5262341bb7c9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:37.570841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:37.571559 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:37.571931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:37.572554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] PATH_INFO: `/v3/users/868dee55d493496090988ff601c7d7c3` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:37.573807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=868dee55d493496090988ff601c7d7c3)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:37.576955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:37.637396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '868dee55d493496090988ff601c7d7c3'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:37.648150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '868dee55d493496090988ff601c7d7c3'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:37.648870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '868dee55d493496090988ff601c7d7c3'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:37.649688 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '868dee55d493496090988ff601c7d7c3'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:37.667324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '868dee55d493496090988ff601c7d7c3'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:37.694522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1079/2142] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:37 2022] DELETE /identity/v3/users/868dee55d493496090988ff601c7d7c3 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:37.696332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e752a59-176a-4087-8f40-9510f50c30ac None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:37.696679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e752a59-176a-4087-8f40-9510f50c30ac None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:37.696958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e752a59-176a-4087-8f40-9510f50c30ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:37.722398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6e752a59-176a-4087-8f40-9510f50c30ac None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:37.723775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6e752a59-176a-4087-8f40-9510f50c30ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:37.827720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1064/2143] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:37.857750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:37.859459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:37.965503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c29825-ce9c-4666-a5aa-4f7de2210a5d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:37.970589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd16ec2c-5af6-4459-99e1-b934ccced36e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:37.971279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16ec2c-5af6-4459-99e1-b934ccced36e service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:37.971600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16ec2c-5af6-4459-99e1-b934ccced36e service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:37.971968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd16ec2c-5af6-4459-99e1-b934ccced36e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:37.987659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd16ec2c-5af6-4459-99e1-b934ccced36e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:37.989113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd16ec2c-5af6-4459-99e1-b934ccced36e service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:38.001458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1080/2144] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:38.563863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e752a59-176a-4087-8f40-9510f50c30ac None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:38.565514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6e752a59-176a-4087-8f40-9510f50c30ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:38.582482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e752a59-176a-4087-8f40-9510f50c30ac None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:38.591644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:38.596143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:38.596486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:38.596798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] PATH_INFO: `/v3/projects/95fe1ad03f21497aad64598e8f6d13f7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:38.598178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95fe1ad03f21497aad64598e8f6d13f7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:38.600411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:38.658212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95fe1ad03f21497aad64598e8f6d13f7', 'request_id': 'req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:38.666924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95fe1ad03f21497aad64598e8f6d13f7', 'request_id': 'req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:38.675622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95fe1ad03f21497aad64598e8f6d13f7', 'request_id': 'req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:38.687768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] The token cache is being invalidate because project 95fe1ad03f21497aad64598e8f6d13f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:38.688241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:38.688902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4d74adbd-a6ff-497f-8d62-f26db8dc9aa1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:38.690379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1065/2145] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:38 2022] DELETE /identity/v3/projects/95fe1ad03f21497aad64598e8f6d13f7 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:38.695402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818380aa-ce28-466f-aa85-44f5167c33d6 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:38.695846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818380aa-ce28-466f-aa85-44f5167c33d6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:38.696191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818380aa-ce28-466f-aa85-44f5167c33d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:38.717667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-818380aa-ce28-466f-aa85-44f5167c33d6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:38.719815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-818380aa-ce28-466f-aa85-44f5167c33d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:38.786574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1081/2146] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:38.795709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb800848-c0d9-4856-b73d-fd4ca62d397e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:38.796062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb800848-c0d9-4856-b73d-fd4ca62d397e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:38.796344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb800848-c0d9-4856-b73d-fd4ca62d397e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:38.819067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-bb800848-c0d9-4856-b73d-fd4ca62d397e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:38.820399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bb800848-c0d9-4856-b73d-fd4ca62d397e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:38.829471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1066/2147] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:38.833725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818380aa-ce28-466f-aa85-44f5167c33d6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:38.844737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818380aa-ce28-466f-aa85-44f5167c33d6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:38.849770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:38.850396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:38.850674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:38.850949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:38.851646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:38.852547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:38.854171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:38.867772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1082/2148] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:11:38.872351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb800848-c0d9-4856-b73d-fd4ca62d397e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:38.883530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb800848-c0d9-4856-b73d-fd4ca62d397e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:38.888800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ef3810-53ff-4175-ac10-23e2ff51cb69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:38.889209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef3810-53ff-4175-ac10-23e2ff51cb69 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:38.889491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef3810-53ff-4175-ac10-23e2ff51cb69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:38.889766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef3810-53ff-4175-ac10-23e2ff51cb69 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:38.890512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ef3810-53ff-4175-ac10-23e2ff51cb69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:38.892462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ef3810-53ff-4175-ac10-23e2ff51cb69 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:38.901562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:38.903383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:38.915343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:38.938273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1067/2149] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:38 2022] POST /identity/v3/projects => generated 397 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:38.939874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ef3810-53ff-4175-ac10-23e2ff51cb69 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:38.947359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcb11ea-6852-447a-9e37-34e440640ea9 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:38.956401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:39.032658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:39.033391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.033900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.034413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] PATH_INFO: `/v3/users/fca684dce8ea4fc48b13f69595289024` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.036020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fca684dce8ea4fc48b13f69595289024)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:39.037887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:39.050796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ef3810-53ff-4175-ac10-23e2ff51cb69 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:39.057734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d84be14-a05e-4939-8b85-83619506640e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:39.058861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d84be14-a05e-4939-8b85-83619506640e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.059324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d84be14-a05e-4939-8b85-83619506640e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.059797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d84be14-a05e-4939-8b85-83619506640e admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.060953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d84be14-a05e-4939-8b85-83619506640e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:39.065208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d84be14-a05e-4939-8b85-83619506640e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:39.101366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1068/2150] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:38 2022] POST /identity/v3/users => generated 299 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:39.105530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d84be14-a05e-4939-8b85-83619506640e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:39.112718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fca684dce8ea4fc48b13f69595289024'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:39.118013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d84be14-a05e-4939-8b85-83619506640e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:39.124554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:39.125098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.126602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.126602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.127158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:39.129229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:39.130258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fca684dce8ea4fc48b13f69595289024'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:39.131632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fca684dce8ea4fc48b13f69595289024'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:39.132938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fca684dce8ea4fc48b13f69595289024'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:39.144796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1069/2151] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:39 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:39.148921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:39.150685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:39.152539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6e9cc7cd-c261-4331-8dbb-9c55eddf6433 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fca684dce8ea4fc48b13f69595289024'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:39.167045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67abfe98-4cf3-4df7-b00f-5a4f75b4fa0c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:39.172876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0310d0d-5de7-4011-8e97-5416953fa83d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:39.173481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0310d0d-5de7-4011-8e97-5416953fa83d admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.173762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0310d0d-5de7-4011-8e97-5416953fa83d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.174055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0310d0d-5de7-4011-8e97-5416953fa83d admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.174703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0310d0d-5de7-4011-8e97-5416953fa83d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:39.176536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0310d0d-5de7-4011-8e97-5416953fa83d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:39.177706 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1083/2152] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:38 2022] DELETE /identity/v3/users/fca684dce8ea4fc48b13f69595289024 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:11:39.179067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0785df89-0847-4f85-841e-061afe9a51ba None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.179652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0785df89-0847-4f85-841e-061afe9a51ba None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.179809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0785df89-0847-4f85-841e-061afe9a51ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.189810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1070/2153] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:39 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:11:39.192935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0310d0d-5de7-4011-8e97-5416953fa83d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:39.201399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-0785df89-0847-4f85-841e-061afe9a51ba None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:39.202602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0785df89-0847-4f85-841e-061afe9a51ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:39.204397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0310d0d-5de7-4011-8e97-5416953fa83d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:39.270654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4c4b22-9c32-4ddb-beec-fe4f585297c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:39.271278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4c4b22-9c32-4ddb-beec-fe4f585297c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.271562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4c4b22-9c32-4ddb-beec-fe4f585297c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.271861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4c4b22-9c32-4ddb-beec-fe4f585297c2 admin admin] PATH_INFO: `/v3/projects/68f5f2734ef54735a5ac845eca45fd8f/users/87d6231de5ae42bab17d11e92def5de7/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.276554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1084/2154] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:39.287247 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0785df89-0847-4f85-841e-061afe9a51ba None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:39.288868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0785df89-0847-4f85-841e-061afe9a51ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:39.289012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4c4b22-9c32-4ddb-beec-fe4f585297c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=68f5f2734ef54735a5ac845eca45fd8f, user_id=87d6231de5ae42bab17d11e92def5de7, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:39.291724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4c4b22-9c32-4ddb-beec-fe4f585297c2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:39.316161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1071/2155] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:39 2022] PUT /identity/v3/projects/68f5f2734ef54735a5ac845eca45fd8f/users/87d6231de5ae42bab17d11e92def5de7/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:11:39.334851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ead2da-b4a5-481b-a922-808273ba0602 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.335160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ead2da-b4a5-481b-a922-808273ba0602 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.335439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ead2da-b4a5-481b-a922-808273ba0602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.342052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:39.367072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-a0ead2da-b4a5-481b-a922-808273ba0602 None None] MFA Rules not processed for user `87d6231de5ae42bab17d11e92def5de7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:39.368319 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a0ead2da-b4a5-481b-a922-808273ba0602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:39.378409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0785df89-0847-4f85-841e-061afe9a51ba None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:39.411798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:39.439663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:39.440433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.440782 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.441129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.448871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1072/2156] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:39 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:39.462041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ead2da-b4a5-481b-a922-808273ba0602 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:39.464346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a0ead2da-b4a5-481b-a922-808273ba0602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:39.515039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:39.518384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:39.530476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1085/2157] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:11:39.533265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ead2da-b4a5-481b-a922-808273ba0602 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:39.540780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ed9ed1-f911-42a7-bd24-edbdf5ab21d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:39.541450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ed9ed1-f911-42a7-bd24-edbdf5ab21d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:39.541771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ed9ed1-f911-42a7-bd24-edbdf5ab21d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:39.542078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ed9ed1-f911-42a7-bd24-edbdf5ab21d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:39.553214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ed9ed1-f911-42a7-bd24-edbdf5ab21d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:39.554815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ed9ed1-f911-42a7-bd24-edbdf5ab21d0 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:39.566421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1073/2158] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:39 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:11:40.038303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:40.039811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:40.065309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d057c38-413c-4ad0-baf9-7402fe3e4e3a service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:40.078114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:40.079563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:40.080049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:40.080330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] PATH_INFO: `/v3/projects/e338eb5f015c4b83904f9783faa1b548` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:40.081476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e338eb5f015c4b83904f9783faa1b548)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:40.084366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:40.139761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e338eb5f015c4b83904f9783faa1b548', 'request_id': 'req-7504126b-1d5a-4835-8be0-faad620c0eb6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.148088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e338eb5f015c4b83904f9783faa1b548', 'request_id': 'req-7504126b-1d5a-4835-8be0-faad620c0eb6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.156368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e338eb5f015c4b83904f9783faa1b548', 'request_id': 'req-7504126b-1d5a-4835-8be0-faad620c0eb6'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.169391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] The token cache is being invalidate because project e338eb5f015c4b83904f9783faa1b548 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:40.169686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.170368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.172442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1086/2159] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:40 2022] DELETE /identity/v3/projects/e338eb5f015c4b83904f9783faa1b548 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:40.176701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ed9ed1-f911-42a7-bd24-edbdf5ab21d0 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:40.188158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ed9ed1-f911-42a7-bd24-edbdf5ab21d0 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:40.247434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:40.248093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:40.248375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:40.248646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] PATH_INFO: `/v3/users/df7f7e9f4e954a198bcfcc52c2a2e7e7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:40.250399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=df7f7e9f4e954a198bcfcc52c2a2e7e7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:40.252434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:40.301317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df7f7e9f4e954a198bcfcc52c2a2e7e7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.316002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df7f7e9f4e954a198bcfcc52c2a2e7e7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.316631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df7f7e9f4e954a198bcfcc52c2a2e7e7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.317347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df7f7e9f4e954a198bcfcc52c2a2e7e7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.324697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d7772c24-b28e-44bc-b0ed-e221094e9f0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df7f7e9f4e954a198bcfcc52c2a2e7e7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.338716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:40.345168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1074/2160] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:40 2022] DELETE /identity/v3/users/df7f7e9f4e954a198bcfcc52c2a2e7e7 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:40.642407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:40.643904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:40.657571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7504126b-1d5a-4835-8be0-faad620c0eb6 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:40.669924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:40.717884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:40.719242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:40.719525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:40.719835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] PATH_INFO: `/v3/projects/ae341e18bba840f3839da2e926f41b17` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:40.720967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae341e18bba840f3839da2e926f41b17)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:40.722913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:40.768916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae341e18bba840f3839da2e926f41b17', 'request_id': 'req-b0edb45d-b5e0-4944-a75c-4cd9e8813365'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.775550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae341e18bba840f3839da2e926f41b17', 'request_id': 'req-b0edb45d-b5e0-4944-a75c-4cd9e8813365'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.781604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae341e18bba840f3839da2e926f41b17', 'request_id': 'req-b0edb45d-b5e0-4944-a75c-4cd9e8813365'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.792466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] The token cache is being invalidate because project ae341e18bba840f3839da2e926f41b17 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:40.792754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.793379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0edb45d-b5e0-4944-a75c-4cd9e8813365 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:40.794847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1087/2161] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:40 2022] DELETE /identity/v3/projects/ae341e18bba840f3839da2e926f41b17 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:40.799810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa10a5d-6376-4673-940e-25372abc5efb None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:40.800300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa10a5d-6376-4673-940e-25372abc5efb None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:40.800393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa10a5d-6376-4673-940e-25372abc5efb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:40.819887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9fa10a5d-6376-4673-940e-25372abc5efb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:40.821006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9fa10a5d-6376-4673-940e-25372abc5efb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:40.878194 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1075/2162] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:40.885923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79568066-3bb0-4bce-b84d-dcf457aa048f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:40.886191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79568066-3bb0-4bce-b84d-dcf457aa048f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:40.886467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79568066-3bb0-4bce-b84d-dcf457aa048f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:40.909463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-79568066-3bb0-4bce-b84d-dcf457aa048f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:40.910675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-79568066-3bb0-4bce-b84d-dcf457aa048f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:40.918684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1088/2163] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:40.924024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa10a5d-6376-4673-940e-25372abc5efb None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:40.933619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa10a5d-6376-4673-940e-25372abc5efb None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:40.938663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:40.939298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:40.939575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:40.939878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:40.940524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:40.940824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:40.942417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:40.954412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1076/2164] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:11:40.958623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79568066-3bb0-4bce-b84d-dcf457aa048f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:40.969272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79568066-3bb0-4bce-b84d-dcf457aa048f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:40.974474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b1d6b5-4fba-41ff-83af-680d7a842def admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:40.975083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b1d6b5-4fba-41ff-83af-680d7a842def admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:40.975384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b1d6b5-4fba-41ff-83af-680d7a842def admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:40.975659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b1d6b5-4fba-41ff-83af-680d7a842def admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:40.976360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b1d6b5-4fba-41ff-83af-680d7a842def admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:40.978343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b1d6b5-4fba-41ff-83af-680d7a842def admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:41.009555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1089/2165] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:40 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:41.010714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:41.022044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377a6fb5-24d2-4c77-992d-2b70b466c7f8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:41.088880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222eedee-148f-4f7e-8b89-5a0c719d53b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:41.089437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222eedee-148f-4f7e-8b89-5a0c719d53b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:41.089826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222eedee-148f-4f7e-8b89-5a0c719d53b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:41.090104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222eedee-148f-4f7e-8b89-5a0c719d53b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:41.090729 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222eedee-148f-4f7e-8b89-5a0c719d53b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:41.092855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222eedee-148f-4f7e-8b89-5a0c719d53b8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:41.116909 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1077/2166] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:41 2022] POST /identity/v3/users => generated 289 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:41.120642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b1d6b5-4fba-41ff-83af-680d7a842def admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:41.132619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b1d6b5-4fba-41ff-83af-680d7a842def admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:41.140119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a89671-c23b-4872-b70d-31dcb2139ad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:41.141433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a89671-c23b-4872-b70d-31dcb2139ad9 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:41.141708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a89671-c23b-4872-b70d-31dcb2139ad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:41.141977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a89671-c23b-4872-b70d-31dcb2139ad9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:41.142628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a89671-c23b-4872-b70d-31dcb2139ad9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:41.144425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a89671-c23b-4872-b70d-31dcb2139ad9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:41.155557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1090/2167] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:41.160207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222eedee-148f-4f7e-8b89-5a0c719d53b8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:41.171149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222eedee-148f-4f7e-8b89-5a0c719d53b8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:41.178733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416317d9-f441-467a-8b9c-5f3442c850bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:41.179343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416317d9-f441-467a-8b9c-5f3442c850bf admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:41.179621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416317d9-f441-467a-8b9c-5f3442c850bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:41.179931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416317d9-f441-467a-8b9c-5f3442c850bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:41.180650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416317d9-f441-467a-8b9c-5f3442c850bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:41.182900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416317d9-f441-467a-8b9c-5f3442c850bf admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:41.206899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1078/2168] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:41 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:11:41.208140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a89671-c23b-4872-b70d-31dcb2139ad9 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:41.220130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a89671-c23b-4872-b70d-31dcb2139ad9 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:41.225034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:41.225746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:41.226032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:41.226313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] PATH_INFO: `/v3/projects/2762041c21af407c82eeb40130ea00f0/users/6ceffbdb03954009bb4b6dd22aba7ac0/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:41.241849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2762041c21af407c82eeb40130ea00f0, user_id=6ceffbdb03954009bb4b6dd22aba7ac0, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:41.244726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:41.259038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1091/2169] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:41 2022] PUT /identity/v3/projects/2762041c21af407c82eeb40130ea00f0/users/6ceffbdb03954009bb4b6dd22aba7ac0/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:41.272029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec43b58a-8f7a-4982-82e3-ff6cbaf9be51 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:41.272320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec43b58a-8f7a-4982-82e3-ff6cbaf9be51 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:41.272598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec43b58a-8f7a-4982-82e3-ff6cbaf9be51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:41.306951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ec43b58a-8f7a-4982-82e3-ff6cbaf9be51 None None] MFA Rules not processed for user `6ceffbdb03954009bb4b6dd22aba7ac0`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:41.308213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ec43b58a-8f7a-4982-82e3-ff6cbaf9be51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:41.379929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:41.392900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1079/2170] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:41 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:41.404588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:41.406842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:41.455631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:41.470958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5a8a37-3a2a-40ae-a809-36ee0a413197 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:41.475646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72265a41-cb2e-4045-b322-df14546c4ed4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:41.476469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72265a41-cb2e-4045-b322-df14546c4ed4 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:41.476755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72265a41-cb2e-4045-b322-df14546c4ed4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:41.477569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72265a41-cb2e-4045-b322-df14546c4ed4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:41.487557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72265a41-cb2e-4045-b322-df14546c4ed4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:41.489227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72265a41-cb2e-4045-b322-df14546c4ed4 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:41.499132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1092/2171] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:41 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:48.815057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec43b58a-8f7a-4982-82e3-ff6cbaf9be51 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:48.816485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ec43b58a-8f7a-4982-82e3-ff6cbaf9be51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:48.879511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec43b58a-8f7a-4982-82e3-ff6cbaf9be51 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:48.883962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:48.884589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:48.884869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:48.885141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] PATH_INFO: `/v3/users/87d6231de5ae42bab17d11e92def5de7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:48.886236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] RBAC: Authorizing `identity:delete_user(user_id=87d6231de5ae42bab17d11e92def5de7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:48.888218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:48.934264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '87d6231de5ae42bab17d11e92def5de7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:48.940851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '87d6231de5ae42bab17d11e92def5de7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:48.941509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '87d6231de5ae42bab17d11e92def5de7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:48.942129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '87d6231de5ae42bab17d11e92def5de7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:48.949109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '87d6231de5ae42bab17d11e92def5de7'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:48.966803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1080/2172] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:48 2022] DELETE /identity/v3/users/87d6231de5ae42bab17d11e92def5de7 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:48.972046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406ee37-420a-460f-91e3-0c4b2a392ce0 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:48.972522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406ee37-420a-460f-91e3-0c4b2a392ce0 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:48.972609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406ee37-420a-460f-91e3-0c4b2a392ce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:48.991109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-f406ee37-420a-460f-91e3-0c4b2a392ce0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:48.992332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f406ee37-420a-460f-91e3-0c4b2a392ce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:49.051578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1093/2173] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:49.061474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:49.063076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:49.119362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd95f47-d125-48ac-aef4-98e7fbbab84e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:49.123064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc1cd54-487c-44af-86bd-8733f12162c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:49.123665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1cd54-487c-44af-86bd-8733f12162c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:49.123981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1cd54-487c-44af-86bd-8733f12162c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:49.124259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1cd54-487c-44af-86bd-8733f12162c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:49.133366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc1cd54-487c-44af-86bd-8733f12162c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:49.134774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc1cd54-487c-44af-86bd-8733f12162c9 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:49.144649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1081/2174] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:49.420087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406ee37-420a-460f-91e3-0c4b2a392ce0 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:49.422482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f406ee37-420a-460f-91e3-0c4b2a392ce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:49.436212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406ee37-420a-460f-91e3-0c4b2a392ce0 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:49.442753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:49.443272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:49.443628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:49.444107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] PATH_INFO: `/v3/projects/68f5f2734ef54735a5ac845eca45fd8f` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:49.445693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=68f5f2734ef54735a5ac845eca45fd8f)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:49.447209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:49.504935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68f5f2734ef54735a5ac845eca45fd8f', 'request_id': 'req-8bc858e9-b468-4cfa-ade5-65b832f229fb'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:49.513114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68f5f2734ef54735a5ac845eca45fd8f', 'request_id': 'req-8bc858e9-b468-4cfa-ade5-65b832f229fb'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:49.519949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68f5f2734ef54735a5ac845eca45fd8f', 'request_id': 'req-8bc858e9-b468-4cfa-ade5-65b832f229fb'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:49.532478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] The token cache is being invalidate because project 68f5f2734ef54735a5ac845eca45fd8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:49.532773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:49.533643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8bc858e9-b468-4cfa-ade5-65b832f229fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:49.535157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1094/2175] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:49 2022] DELETE /identity/v3/projects/68f5f2734ef54735a5ac845eca45fd8f => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:57.540744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb510b-19ee-4550-8097-6cbdf12b909a None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:57.541344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb510b-19ee-4550-8097-6cbdf12b909a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:57.541344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb510b-19ee-4550-8097-6cbdf12b909a None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:57.542631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1082/2176] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:11:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:11:57.554481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216c9398-f7be-41d8-a1c9-8bbc67a154f1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:57.554957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216c9398-f7be-41d8-a1c9-8bbc67a154f1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:57.555070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216c9398-f7be-41d8-a1c9-8bbc67a154f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:57.575470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-216c9398-f7be-41d8-a1c9-8bbc67a154f1 None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:57.577029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-216c9398-f7be-41d8-a1c9-8bbc67a154f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:57.620868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb510b-19ee-4550-8097-6cbdf12b909a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:57.624728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c9fb510b-19ee-4550-8097-6cbdf12b909a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:57.645851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:57.645996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1095/2177] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:11:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:57.658122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216c9398-f7be-41d8-a1c9-8bbc67a154f1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:57.659485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-216c9398-f7be-41d8-a1c9-8bbc67a154f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:57.668683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:57.673990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216c9398-f7be-41d8-a1c9-8bbc67a154f1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:57.683915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bf2fba-b09e-470b-b129-ddaca6d17bb7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:57.684792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf2fba-b09e-470b-b129-ddaca6d17bb7 service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:57.685071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf2fba-b09e-470b-b129-ddaca6d17bb7 service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:57.685467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf2fba-b09e-470b-b129-ddaca6d17bb7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:57.694711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bf2fba-b09e-470b-b129-ddaca6d17bb7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:57.696362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bf2fba-b09e-470b-b129-ddaca6d17bb7 service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:57.701048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb510b-19ee-4550-8097-6cbdf12b909a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:57.705813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:57.706313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1096/2178] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:11:57.706488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:57.706706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:57.706980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] PATH_INFO: `/v3/users/f521b038310940d1b5ee7fc5afc1cbac` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:57.708144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f521b038310940d1b5ee7fc5afc1cbac)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:57.710048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:57.757176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f521b038310940d1b5ee7fc5afc1cbac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:57.764091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f521b038310940d1b5ee7fc5afc1cbac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:57.764722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f521b038310940d1b5ee7fc5afc1cbac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:57.765492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f521b038310940d1b5ee7fc5afc1cbac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:57.772437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-682a3693-9c4c-4724-971c-62c4e448162d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f521b038310940d1b5ee7fc5afc1cbac'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:57.790019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1083/2179] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:11:57 2022] DELETE /identity/v3/users/f521b038310940d1b5ee7fc5afc1cbac => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:57.958910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bf2fba-b09e-470b-b129-ddaca6d17bb7 service placement] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:57.980369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bf2fba-b09e-470b-b129-ddaca6d17bb7 service placement] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.040249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.041026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.041399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.041746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] PATH_INFO: `/v3/projects/d903de84a9c7446e9a2f368e5eab8719` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.042923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d903de84a9c7446e9a2f368e5eab8719)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.045086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.092012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd903de84a9c7446e9a2f368e5eab8719', 'request_id': 'req-c1568769-2e95-46c6-994c-01b4ed24003e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:58.100193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd903de84a9c7446e9a2f368e5eab8719', 'request_id': 'req-c1568769-2e95-46c6-994c-01b4ed24003e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:58.106078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd903de84a9c7446e9a2f368e5eab8719', 'request_id': 'req-c1568769-2e95-46c6-994c-01b4ed24003e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:58.119182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] The token cache is being invalidate because project d903de84a9c7446e9a2f368e5eab8719 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:11:58.119481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:58.120180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c1568769-2e95-46c6-994c-01b4ed24003e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:11:58.121694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1097/2180] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:11:57 2022] DELETE /identity/v3/projects/d903de84a9c7446e9a2f368e5eab8719 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:58.132833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd2a2dc-73fc-4f8c-aad4-0faf4ed56d7a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.133231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd2a2dc-73fc-4f8c-aad4-0faf4ed56d7a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.133517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd2a2dc-73fc-4f8c-aad4-0faf4ed56d7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.154624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-dfd2a2dc-73fc-4f8c-aad4-0faf4ed56d7a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:58.156514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dfd2a2dc-73fc-4f8c-aad4-0faf4ed56d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:58.227811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1084/2181] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:11:58.231571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52ba421-d625-496c-b400-cf0569f78de5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.232013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52ba421-d625-496c-b400-cf0569f78de5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.232360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52ba421-d625-496c-b400-cf0569f78de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.257427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a52ba421-d625-496c-b400-cf0569f78de5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:58.258746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a52ba421-d625-496c-b400-cf0569f78de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:58.271334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1098/2182] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:58.275730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd2a2dc-73fc-4f8c-aad4-0faf4ed56d7a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.286316 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd2a2dc-73fc-4f8c-aad4-0faf4ed56d7a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.291003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.291625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.291952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.292224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.292881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:11:58.293180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.294905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.315127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1085/2183] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:11:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:11:58.319472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52ba421-d625-496c-b400-cf0569f78de5 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.329389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52ba421-d625-496c-b400-cf0569f78de5 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.334178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f36ce3-85a7-4b32-be49-0a7615ff5c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.334780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f36ce3-85a7-4b32-be49-0a7615ff5c00 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.335059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f36ce3-85a7-4b32-be49-0a7615ff5c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.335330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f36ce3-85a7-4b32-be49-0a7615ff5c00 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.336058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f36ce3-85a7-4b32-be49-0a7615ff5c00 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.337885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f36ce3-85a7-4b32-be49-0a7615ff5c00 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.367065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1099/2184] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:58 2022] POST /identity/v3/projects => generated 369 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:58.371241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.381751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54627f1d-fe13-4a8d-b38b-0d3bd508ad8a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.401749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:58.443319 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f8fec3-2549-473e-8629-6adb032b42e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.443960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f8fec3-2549-473e-8629-6adb032b42e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.444240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f8fec3-2549-473e-8629-6adb032b42e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.444512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f8fec3-2549-473e-8629-6adb032b42e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.445191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f8fec3-2549-473e-8629-6adb032b42e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.447103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f8fec3-2549-473e-8629-6adb032b42e4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.472481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1086/2185] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:58 2022] POST /identity/v3/users => generated 285 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:58.473682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f36ce3-85a7-4b32-be49-0a7615ff5c00 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.485705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f36ce3-85a7-4b32-be49-0a7615ff5c00 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.491351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a354c97-14b5-4fac-8185-96c48f5ee2ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.491998 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a354c97-14b5-4fac-8185-96c48f5ee2ec admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.492282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a354c97-14b5-4fac-8185-96c48f5ee2ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.492556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a354c97-14b5-4fac-8185-96c48f5ee2ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.493202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a354c97-14b5-4fac-8185-96c48f5ee2ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.494811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a354c97-14b5-4fac-8185-96c48f5ee2ec admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.505621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1100/2186] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:58.509503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f8fec3-2549-473e-8629-6adb032b42e4 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.520197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f8fec3-2549-473e-8629-6adb032b42e4 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.525248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6208261-b6bf-4276-8d36-3d4d05aaa306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.525871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6208261-b6bf-4276-8d36-3d4d05aaa306 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.526163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6208261-b6bf-4276-8d36-3d4d05aaa306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.526443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6208261-b6bf-4276-8d36-3d4d05aaa306 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.527102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6208261-b6bf-4276-8d36-3d4d05aaa306 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.528752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6208261-b6bf-4276-8d36-3d4d05aaa306 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.541806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1087/2187] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:58 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:58.545638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a354c97-14b5-4fac-8185-96c48f5ee2ec admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.556261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a354c97-14b5-4fac-8185-96c48f5ee2ec admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.567076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff27b4c-04b4-4d33-9759-8af362062c2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.567795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff27b4c-04b4-4d33-9759-8af362062c2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.568079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff27b4c-04b4-4d33-9759-8af362062c2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.568356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff27b4c-04b4-4d33-9759-8af362062c2e admin admin] PATH_INFO: `/v3/projects/e2ebcaaf34c84d33bf480243fe212896/users/7f589ccb997b46b3b6fcfc4925254e5a/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.584016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff27b4c-04b4-4d33-9759-8af362062c2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2ebcaaf34c84d33bf480243fe212896, user_id=7f589ccb997b46b3b6fcfc4925254e5a, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.586686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff27b4c-04b4-4d33-9759-8af362062c2e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.610846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1101/2188] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:58 2022] PUT /identity/v3/projects/e2ebcaaf34c84d33bf480243fe212896/users/7f589ccb997b46b3b6fcfc4925254e5a/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:58.622421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a850fb-6c4b-4c45-a44d-41b2ebba0842 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.622703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a850fb-6c4b-4c45-a44d-41b2ebba0842 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.622985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a850fb-6c4b-4c45-a44d-41b2ebba0842 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.652798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-40a850fb-6c4b-4c45-a44d-41b2ebba0842 None None] MFA Rules not processed for user `7f589ccb997b46b3b6fcfc4925254e5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:58.654010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-40a850fb-6c4b-4c45-a44d-41b2ebba0842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:58.724268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1088/2189] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:58 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:58.728821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff27b4c-04b4-4d33-9759-8af362062c2e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.748885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:58.795535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff27b4c-04b4-4d33-9759-8af362062c2e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.800424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dc6f5f-96fe-4d89-b817-a9442db72ca0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.801033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dc6f5f-96fe-4d89-b817-a9442db72ca0 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.801318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dc6f5f-96fe-4d89-b817-a9442db72ca0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.801596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dc6f5f-96fe-4d89-b817-a9442db72ca0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.802246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dc6f5f-96fe-4d89-b817-a9442db72ca0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.804215 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dc6f5f-96fe-4d89-b817-a9442db72ca0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.837195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a850fb-6c4b-4c45-a44d-41b2ebba0842 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.843240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1102/2190] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:11:58 2022] POST /identity/v3/projects => generated 369 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:11:58.849918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a850fb-6c4b-4c45-a44d-41b2ebba0842 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.910653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10afa6b-f29a-478a-b7dc-9f7749dbd565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.911272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10afa6b-f29a-478a-b7dc-9f7749dbd565 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.911559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10afa6b-f29a-478a-b7dc-9f7749dbd565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.912022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10afa6b-f29a-478a-b7dc-9f7749dbd565 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.912672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10afa6b-f29a-478a-b7dc-9f7749dbd565 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.914585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10afa6b-f29a-478a-b7dc-9f7749dbd565 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.936701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1089/2191] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:11:58 2022] POST /identity/v3/users => generated 291 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:11:58.941303 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dc6f5f-96fe-4d89-b817-a9442db72ca0 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.951533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dc6f5f-96fe-4d89-b817-a9442db72ca0 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.956651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e687f-4f19-4d2a-a13c-12632950c652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.957276 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e687f-4f19-4d2a-a13c-12632950c652 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.957557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e687f-4f19-4d2a-a13c-12632950c652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.957834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e687f-4f19-4d2a-a13c-12632950c652 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:58.958545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9e687f-4f19-4d2a-a13c-12632950c652 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:58.960224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9e687f-4f19-4d2a-a13c-12632950c652 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:58.971898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1103/2192] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:58.976313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10afa6b-f29a-478a-b7dc-9f7749dbd565 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:58.987277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10afa6b-f29a-478a-b7dc-9f7749dbd565 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:58.991972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d6835-6200-4b31-9489-639b9b90a56d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:58.992653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d6835-6200-4b31-9489-639b9b90a56d admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:58.993099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d6835-6200-4b31-9489-639b9b90a56d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:58.993379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d6835-6200-4b31-9489-639b9b90a56d admin admin] PATH_INFO: `/v3/projects/9b989c6542ae4336a0c1ee1e468aed38/users/1e4c9929f12b4e669160cbb74cb10775/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:59.012788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0d6835-6200-4b31-9489-639b9b90a56d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b989c6542ae4336a0c1ee1e468aed38, user_id=1e4c9929f12b4e669160cbb74cb10775, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:59.015460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0d6835-6200-4b31-9489-639b9b90a56d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:59.018921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:59.031445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1090/2193] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:58 2022] PUT /identity/v3/projects/9b989c6542ae4336a0c1ee1e468aed38/users/1e4c9929f12b4e669160cbb74cb10775/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:59.035630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e687f-4f19-4d2a-a13c-12632950c652 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:59.046168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e687f-4f19-4d2a-a13c-12632950c652 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:59.106510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8946cdb2-f8fb-40a5-9a81-021b7093cd89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:59.107233 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8946cdb2-f8fb-40a5-9a81-021b7093cd89 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:59.107498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8946cdb2-f8fb-40a5-9a81-021b7093cd89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:59.107794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8946cdb2-f8fb-40a5-9a81-021b7093cd89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:59.108445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8946cdb2-f8fb-40a5-9a81-021b7093cd89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:59.111263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8946cdb2-f8fb-40a5-9a81-021b7093cd89 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:59.121575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1104/2194] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:11:59 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:11:59.125958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d6835-6200-4b31-9489-639b9b90a56d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:59.136818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d6835-6200-4b31-9489-639b9b90a56d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:59.141795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:59.142463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:59.142740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:59.143009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] PATH_INFO: `/v3/projects/9b989c6542ae4336a0c1ee1e468aed38/users/1e4c9929f12b4e669160cbb74cb10775/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:59.145001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b989c6542ae4336a0c1ee1e468aed38, user_id=1e4c9929f12b4e669160cbb74cb10775, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:59.147670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:59.159900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1091/2195] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:11:59 2022] PUT /identity/v3/projects/9b989c6542ae4336a0c1ee1e468aed38/users/1e4c9929f12b4e669160cbb74cb10775/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:11:59.172767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f939f2-8f12-4735-afe3-533389fa14f0 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:59.173324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f939f2-8f12-4735-afe3-533389fa14f0 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:59.173599 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f939f2-8f12-4735-afe3-533389fa14f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:59.203242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-77f939f2-8f12-4735-afe3-533389fa14f0 None None] MFA Rules not processed for user `1e4c9929f12b4e669160cbb74cb10775`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:11:59.204508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-77f939f2-8f12-4735-afe3-533389fa14f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:59.275065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1105/2196] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:11:59 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:11:59.288768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:59.291452 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:59.352848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65aaf130-59d6-4f11-8579-d03d3a48aaee admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:59.357794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b243e765-ab55-403d-93a7-0a184cb4d2a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:59.358407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b243e765-ab55-403d-93a7-0a184cb4d2a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:59.358689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b243e765-ab55-403d-93a7-0a184cb4d2a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:59.359603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b243e765-ab55-403d-93a7-0a184cb4d2a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:59.371363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b243e765-ab55-403d-93a7-0a184cb4d2a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:59.373261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b243e765-ab55-403d-93a7-0a184cb4d2a4 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:59.385612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1092/2197] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:59 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:11:59.437236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f939f2-8f12-4735-afe3-533389fa14f0 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:11:59.438944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-77f939f2-8f12-4735-afe3-533389fa14f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:11:59.450825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f939f2-8f12-4735-afe3-533389fa14f0 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:11:59.455312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be268c50-708b-46a2-af60-4f833953056b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:11:59.455959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be268c50-708b-46a2-af60-4f833953056b service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:11:59.456237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be268c50-708b-46a2-af60-4f833953056b service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:11:59.456506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be268c50-708b-46a2-af60-4f833953056b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:11:59.467323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:11:59.527836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be268c50-708b-46a2-af60-4f833953056b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:11:59.529323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be268c50-708b-46a2-af60-4f833953056b service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:11:59.540691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1106/2198] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:11:59 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:02.958951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b243e765-ab55-403d-93a7-0a184cb4d2a4 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:03.029969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b243e765-ab55-403d-93a7-0a184cb4d2a4 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:03.034647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:03.035285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:03.035569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:03.035910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] PATH_INFO: `/v3/users/7f589ccb997b46b3b6fcfc4925254e5a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:03.037079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f589ccb997b46b3b6fcfc4925254e5a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:03.039029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:03.091014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f589ccb997b46b3b6fcfc4925254e5a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.098879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f589ccb997b46b3b6fcfc4925254e5a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.099519 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f589ccb997b46b3b6fcfc4925254e5a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.100351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f589ccb997b46b3b6fcfc4925254e5a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.108357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f589ccb997b46b3b6fcfc4925254e5a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.126601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1093/2199] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:02 2022] DELETE /identity/v3/users/7f589ccb997b46b3b6fcfc4925254e5a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:03.131238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37005bdc-bbd0-45bd-91a6-efe06109bb83 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:03.131732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37005bdc-bbd0-45bd-91a6-efe06109bb83 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:03.131863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37005bdc-bbd0-45bd-91a6-efe06109bb83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:03.151878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-37005bdc-bbd0-45bd-91a6-efe06109bb83 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:03.153080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-37005bdc-bbd0-45bd-91a6-efe06109bb83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:03.214751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1107/2200] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:12:03.225080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:03.226614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:03.233592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:03.291597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6b79c4-635d-4f92-8c4e-a69f34c97457 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:03.295180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bc1f02-1266-417e-a031-fcebbd8ba4ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:03.295816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bc1f02-1266-417e-a031-fcebbd8ba4ca service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:03.296096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bc1f02-1266-417e-a031-fcebbd8ba4ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:03.296366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bc1f02-1266-417e-a031-fcebbd8ba4ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:03.310880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bc1f02-1266-417e-a031-fcebbd8ba4ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:03.312586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bc1f02-1266-417e-a031-fcebbd8ba4ca service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:03.323495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1094/2201] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:12:03.606869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37005bdc-bbd0-45bd-91a6-efe06109bb83 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:03.608351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-37005bdc-bbd0-45bd-91a6-efe06109bb83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:03.621705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37005bdc-bbd0-45bd-91a6-efe06109bb83 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:03.629892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:03.630674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:03.630955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:03.631228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] PATH_INFO: `/v3/projects/e2ebcaaf34c84d33bf480243fe212896` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:03.632482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2ebcaaf34c84d33bf480243fe212896)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:03.634415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:03.693468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2ebcaaf34c84d33bf480243fe212896', 'request_id': 'req-3eb80387-0959-4d57-8ff4-62e05d5614a0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.702504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2ebcaaf34c84d33bf480243fe212896', 'request_id': 'req-3eb80387-0959-4d57-8ff4-62e05d5614a0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.709741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2ebcaaf34c84d33bf480243fe212896', 'request_id': 'req-3eb80387-0959-4d57-8ff4-62e05d5614a0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.723741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] The token cache is being invalidate because project e2ebcaaf34c84d33bf480243fe212896 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:03.724008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.724786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.726294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1108/2202] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:03 2022] DELETE /identity/v3/projects/e2ebcaaf34c84d33bf480243fe212896 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:03.732881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bc1f02-1266-417e-a031-fcebbd8ba4ca service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:03.745910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bc1f02-1266-417e-a031-fcebbd8ba4ca service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:03.808035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:03.808659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:03.808940 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:03.809483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] PATH_INFO: `/v3/users/1e4c9929f12b4e669160cbb74cb10775` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:03.810601 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e4c9929f12b4e669160cbb74cb10775)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:03.812551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:03.858262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e4c9929f12b4e669160cbb74cb10775'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.865798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e4c9929f12b4e669160cbb74cb10775'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.866585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e4c9929f12b4e669160cbb74cb10775'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.867177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e4c9929f12b4e669160cbb74cb10775'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.874152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-6f7fb14c-7959-4152-902a-6de6330b7908 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e4c9929f12b4e669160cbb74cb10775'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:03.894170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1095/2203] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:03 2022] DELETE /identity/v3/users/1e4c9929f12b4e669160cbb74cb10775 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:04.175708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:04.177148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:04.192028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb80387-0959-4d57-8ff4-62e05d5614a0 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:04.227759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:04.255677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:04.256257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.256538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.256810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] PATH_INFO: `/v3/projects/9b989c6542ae4336a0c1ee1e468aed38` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.257935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b989c6542ae4336a0c1ee1e468aed38)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:04.259902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:04.323573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b989c6542ae4336a0c1ee1e468aed38', 'request_id': 'req-d2febb6d-3282-49b2-8309-3b215f83cffd'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:04.330897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b989c6542ae4336a0c1ee1e468aed38', 'request_id': 'req-d2febb6d-3282-49b2-8309-3b215f83cffd'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:04.336563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b989c6542ae4336a0c1ee1e468aed38', 'request_id': 'req-d2febb6d-3282-49b2-8309-3b215f83cffd'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:04.348440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] The token cache is being invalidate because project 9b989c6542ae4336a0c1ee1e468aed38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:04.348728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:04.349367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d2febb6d-3282-49b2-8309-3b215f83cffd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:04.350844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1109/2204] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:04 2022] DELETE /identity/v3/projects/9b989c6542ae4336a0c1ee1e468aed38 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:04.355650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e14a3-6534-4d5c-8769-742c2df4ee40 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.356142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e14a3-6534-4d5c-8769-742c2df4ee40 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.356236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203e14a3-6534-4d5c-8769-742c2df4ee40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.376846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-203e14a3-6534-4d5c-8769-742c2df4ee40 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:04.378024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-203e14a3-6534-4d5c-8769-742c2df4ee40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:04.398142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:04.437956 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1096/2205] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:04.445766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284b0c1-0431-4398-a321-65569f2f07a5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.446054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284b0c1-0431-4398-a321-65569f2f07a5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.446448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3284b0c1-0431-4398-a321-65569f2f07a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.465700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3284b0c1-0431-4398-a321-65569f2f07a5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:04.466913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3284b0c1-0431-4398-a321-65569f2f07a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:04.475520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1110/2206] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:04.479620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203e14a3-6534-4d5c-8769-742c2df4ee40 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:04.490046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203e14a3-6534-4d5c-8769-742c2df4ee40 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:04.494644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:04.495252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.495528 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.495853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.496504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:12:04.496804 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:04.498393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:04.510743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1097/2207] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:12:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:12:04.515287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3284b0c1-0431-4398-a321-65569f2f07a5 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:04.525263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3284b0c1-0431-4398-a321-65569f2f07a5 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:04.530131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2849353b-3a70-410b-b6b6-0e00962fa87b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:04.530723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2849353b-3a70-410b-b6b6-0e00962fa87b admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.530998 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2849353b-3a70-410b-b6b6-0e00962fa87b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.531267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2849353b-3a70-410b-b6b6-0e00962fa87b admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.532893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2849353b-3a70-410b-b6b6-0e00962fa87b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:04.534814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2849353b-3a70-410b-b6b6-0e00962fa87b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:04.562853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1111/2208] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:04 2022] POST /identity/v3/projects => generated 389 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:04.567194 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:04.577074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ff867-6c09-4c0c-80d1-29102af9618b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:04.635419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f89680-02a0-4b6b-bff1-11de0f7d266b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:04.636065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f89680-02a0-4b6b-bff1-11de0f7d266b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.636349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f89680-02a0-4b6b-bff1-11de0f7d266b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.636626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f89680-02a0-4b6b-bff1-11de0f7d266b admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.637260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f89680-02a0-4b6b-bff1-11de0f7d266b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:04.639180 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f89680-02a0-4b6b-bff1-11de0f7d266b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:04.662230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1098/2209] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:04 2022] POST /identity/v3/users => generated 295 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:04.666492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2849353b-3a70-410b-b6b6-0e00962fa87b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:04.676655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2849353b-3a70-410b-b6b6-0e00962fa87b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:04.681267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f40de0-f7a7-4487-9b83-781205599a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:04.681856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f40de0-f7a7-4487-9b83-781205599a01 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.682200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f40de0-f7a7-4487-9b83-781205599a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.682417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f40de0-f7a7-4487-9b83-781205599a01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.683053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f40de0-f7a7-4487-9b83-781205599a01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:04.684940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f40de0-f7a7-4487-9b83-781205599a01 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:04.694633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1112/2210] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:04.698515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f89680-02a0-4b6b-bff1-11de0f7d266b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:04.708887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f89680-02a0-4b6b-bff1-11de0f7d266b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:04.713580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a51b01f-6e50-4d43-8ef1-7ed0889abe9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:04.714186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51b01f-6e50-4d43-8ef1-7ed0889abe9d admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.714464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51b01f-6e50-4d43-8ef1-7ed0889abe9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.714735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51b01f-6e50-4d43-8ef1-7ed0889abe9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.715381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a51b01f-6e50-4d43-8ef1-7ed0889abe9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:04.717027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a51b01f-6e50-4d43-8ef1-7ed0889abe9d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:04.728608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1099/2211] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:04.732981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f40de0-f7a7-4487-9b83-781205599a01 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:04.743644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f40de0-f7a7-4487-9b83-781205599a01 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:04.748232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c01f70b-04dd-4f00-bf4a-31cc875003fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:04.748883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c01f70b-04dd-4f00-bf4a-31cc875003fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.749165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c01f70b-04dd-4f00-bf4a-31cc875003fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.749441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c01f70b-04dd-4f00-bf4a-31cc875003fc admin admin] PATH_INFO: `/v3/projects/854965d1749f46a0b5df246a8caa8789/users/fdf72082dbaf4573a4c6314109c6ef8d/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.764099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c01f70b-04dd-4f00-bf4a-31cc875003fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=854965d1749f46a0b5df246a8caa8789, user_id=fdf72082dbaf4573a4c6314109c6ef8d, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:04.767418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c01f70b-04dd-4f00-bf4a-31cc875003fc admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:04.781232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1113/2212] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:04 2022] PUT /identity/v3/projects/854965d1749f46a0b5df246a8caa8789/users/fdf72082dbaf4573a4c6314109c6ef8d/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:04.793747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c519a8-c121-40d6-aec2-e80af47aa9a0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.794034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c519a8-c121-40d6-aec2-e80af47aa9a0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.794308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c519a8-c121-40d6-aec2-e80af47aa9a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.830045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f8c519a8-c121-40d6-aec2-e80af47aa9a0 None None] MFA Rules not processed for user `fdf72082dbaf4573a4c6314109c6ef8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:04.831249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f8c519a8-c121-40d6-aec2-e80af47aa9a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:04.898979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1100/2213] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:04 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:12:04.903496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c01f70b-04dd-4f00-bf4a-31cc875003fc admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:04.968503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c01f70b-04dd-4f00-bf4a-31cc875003fc admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:04.973797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5f327a-a73c-49a0-8e77-53126892c842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:04.974393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5f327a-a73c-49a0-8e77-53126892c842 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:04.974668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5f327a-a73c-49a0-8e77-53126892c842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:04.974936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5f327a-a73c-49a0-8e77-53126892c842 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:04.975565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5f327a-a73c-49a0-8e77-53126892c842 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:04.977528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5f327a-a73c-49a0-8e77-53126892c842 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:04.982785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:05.011070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1114/2214] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:04 2022] POST /identity/v3/projects => generated 389 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:05.012313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c519a8-c121-40d6-aec2-e80af47aa9a0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:05.025462 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c519a8-c121-40d6-aec2-e80af47aa9a0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:05.044660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:05.087706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b538a092-9cc1-4a6c-a1bd-94575840dd0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:05.088399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b538a092-9cc1-4a6c-a1bd-94575840dd0e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.088673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b538a092-9cc1-4a6c-a1bd-94575840dd0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.088944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b538a092-9cc1-4a6c-a1bd-94575840dd0e admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.089579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b538a092-9cc1-4a6c-a1bd-94575840dd0e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:05.091479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b538a092-9cc1-4a6c-a1bd-94575840dd0e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:05.116134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1101/2215] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:05 2022] POST /identity/v3/users => generated 301 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:12:05.120149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5f327a-a73c-49a0-8e77-53126892c842 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:05.130760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5f327a-a73c-49a0-8e77-53126892c842 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:05.135543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59640c0d-95d0-4d9a-88e6-5f7a133ad5ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:05.136154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59640c0d-95d0-4d9a-88e6-5f7a133ad5ee admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.136432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59640c0d-95d0-4d9a-88e6-5f7a133ad5ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.136698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59640c0d-95d0-4d9a-88e6-5f7a133ad5ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.137359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59640c0d-95d0-4d9a-88e6-5f7a133ad5ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:05.138932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59640c0d-95d0-4d9a-88e6-5f7a133ad5ee admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:05.152021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1115/2216] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:05.156237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b538a092-9cc1-4a6c-a1bd-94575840dd0e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:05.169275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b538a092-9cc1-4a6c-a1bd-94575840dd0e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:05.176610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb7a45d-c7c1-4d1f-9e73-b493839d39be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:05.177480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7a45d-c7c1-4d1f-9e73-b493839d39be admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.177761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7a45d-c7c1-4d1f-9e73-b493839d39be admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.178038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7a45d-c7c1-4d1f-9e73-b493839d39be admin admin] PATH_INFO: `/v3/projects/8fc5873505c64410a64175e5408f9093/users/3fb8810129ac450ba8c264495b43e9ab/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.194191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb7a45d-c7c1-4d1f-9e73-b493839d39be admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fc5873505c64410a64175e5408f9093, user_id=3fb8810129ac450ba8c264495b43e9ab, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:05.196901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb7a45d-c7c1-4d1f-9e73-b493839d39be admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:05.210856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1102/2217] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:05 2022] PUT /identity/v3/projects/8fc5873505c64410a64175e5408f9093/users/3fb8810129ac450ba8c264495b43e9ab/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:05.215054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59640c0d-95d0-4d9a-88e6-5f7a133ad5ee admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:05.225496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59640c0d-95d0-4d9a-88e6-5f7a133ad5ee admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:05.282535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e238a0d-f2e0-4ccc-a515-44f22e1019f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:05.283062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e238a0d-f2e0-4ccc-a515-44f22e1019f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.283347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e238a0d-f2e0-4ccc-a515-44f22e1019f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.283626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e238a0d-f2e0-4ccc-a515-44f22e1019f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.284304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e238a0d-f2e0-4ccc-a515-44f22e1019f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:05.285912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e238a0d-f2e0-4ccc-a515-44f22e1019f2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:05.295485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1116/2218] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:05 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:05.300253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb7a45d-c7c1-4d1f-9e73-b493839d39be admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:05.313837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb7a45d-c7c1-4d1f-9e73-b493839d39be admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:05.318886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:05.319550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.319873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.320162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] PATH_INFO: `/v3/projects/8fc5873505c64410a64175e5408f9093/users/3fb8810129ac450ba8c264495b43e9ab/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.322096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fc5873505c64410a64175e5408f9093, user_id=3fb8810129ac450ba8c264495b43e9ab, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:05.324851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:05.337392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1103/2219] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:05 2022] PUT /identity/v3/projects/8fc5873505c64410a64175e5408f9093/users/3fb8810129ac450ba8c264495b43e9ab/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:05.348320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67eed3d-a6c2-42fc-b87b-93976962915c None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.348853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67eed3d-a6c2-42fc-b87b-93976962915c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.349133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67eed3d-a6c2-42fc-b87b-93976962915c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.388302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d67eed3d-a6c2-42fc-b87b-93976962915c None None] MFA Rules not processed for user `3fb8810129ac450ba8c264495b43e9ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:05.389525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d67eed3d-a6c2-42fc-b87b-93976962915c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:05.463442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1117/2220] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:05 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:05.474144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:05.476235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:05.536454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ef503a-a362-4697-9d74-e8bdc0aa0ff8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:05.539962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80201d7-7074-4c31-8fec-67ffe0ab6621 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:05.540554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80201d7-7074-4c31-8fec-67ffe0ab6621 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.540832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80201d7-7074-4c31-8fec-67ffe0ab6621 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.541103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80201d7-7074-4c31-8fec-67ffe0ab6621 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.550779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80201d7-7074-4c31-8fec-67ffe0ab6621 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:05.552231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80201d7-7074-4c31-8fec-67ffe0ab6621 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:05.561655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1104/2221] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:05 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:05.700641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67eed3d-a6c2-42fc-b87b-93976962915c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:05.758013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:05.762207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67eed3d-a6c2-42fc-b87b-93976962915c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:05.767734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:05.768365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.768647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.768923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] PATH_INFO: `/v3/users/fdf72082dbaf4573a4c6314109c6ef8d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.770045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdf72082dbaf4573a4c6314109c6ef8d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:05.772062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:05.820579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdf72082dbaf4573a4c6314109c6ef8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:05.834000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdf72082dbaf4573a4c6314109c6ef8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:05.834616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdf72082dbaf4573a4c6314109c6ef8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:05.835220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdf72082dbaf4573a4c6314109c6ef8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:05.842804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdf72082dbaf4573a4c6314109c6ef8d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:05.860792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1118/2222] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:05 2022] DELETE /identity/v3/users/fdf72082dbaf4573a4c6314109c6ef8d => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:05.865176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe4c3b2-b0f6-4395-9a50-f4271ae00621 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:05.865484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe4c3b2-b0f6-4395-9a50-f4271ae00621 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:05.865758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe4c3b2-b0f6-4395-9a50-f4271ae00621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:05.884545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-dfe4c3b2-b0f6-4395-9a50-f4271ae00621 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:05.885749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dfe4c3b2-b0f6-4395-9a50-f4271ae00621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:05.944323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1105/2223] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:05.954172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:05.955584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:06.014609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499a467a-52ad-405a-b7c5-0d77219a63ae admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:06.018377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877b2a9b-1295-45aa-9144-ae59989b29ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:06.018964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877b2a9b-1295-45aa-9144-ae59989b29ad service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:06.019249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877b2a9b-1295-45aa-9144-ae59989b29ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:06.019524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877b2a9b-1295-45aa-9144-ae59989b29ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:06.028436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877b2a9b-1295-45aa-9144-ae59989b29ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:06.029867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877b2a9b-1295-45aa-9144-ae59989b29ad service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:06.039988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1119/2224] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:06.313211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe4c3b2-b0f6-4395-9a50-f4271ae00621 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:06.314769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dfe4c3b2-b0f6-4395-9a50-f4271ae00621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:06.331935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe4c3b2-b0f6-4395-9a50-f4271ae00621 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:06.336649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:06.337353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:06.337702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:06.337979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] PATH_INFO: `/v3/projects/854965d1749f46a0b5df246a8caa8789` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:06.339124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=854965d1749f46a0b5df246a8caa8789)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:06.341061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:06.353364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:06.390099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '854965d1749f46a0b5df246a8caa8789', 'request_id': 'req-94126df6-0aef-4ff7-8772-b70d31ca7c1c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.397754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '854965d1749f46a0b5df246a8caa8789', 'request_id': 'req-94126df6-0aef-4ff7-8772-b70d31ca7c1c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.404626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '854965d1749f46a0b5df246a8caa8789', 'request_id': 'req-94126df6-0aef-4ff7-8772-b70d31ca7c1c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.416558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] The token cache is being invalidate because project 854965d1749f46a0b5df246a8caa8789 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:06.417184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.417532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.419093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1106/2225] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:06 2022] DELETE /identity/v3/projects/854965d1749f46a0b5df246a8caa8789 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:06.423157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877b2a9b-1295-45aa-9144-ae59989b29ad service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:06.432759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877b2a9b-1295-45aa-9144-ae59989b29ad service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:06.490236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:06.490875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:06.491183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:06.491463 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] PATH_INFO: `/v3/users/3fb8810129ac450ba8c264495b43e9ab` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:06.492642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3fb8810129ac450ba8c264495b43e9ab)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:06.494608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:06.516991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:06.544101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3fb8810129ac450ba8c264495b43e9ab'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.553501 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3fb8810129ac450ba8c264495b43e9ab'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.554182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3fb8810129ac450ba8c264495b43e9ab'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.554918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3fb8810129ac450ba8c264495b43e9ab'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.563175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a0ca61ec-42d1-45f6-95c5-a1153b63ca0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3fb8810129ac450ba8c264495b43e9ab'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.582438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1120/2226] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:06 2022] DELETE /identity/v3/users/3fb8810129ac450ba8c264495b43e9ab => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:06.851739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:06.853787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:06.866882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94126df6-0aef-4ff7-8772-b70d31ca7c1c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:06.925603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:06.926147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:06.926432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:06.926710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] PATH_INFO: `/v3/projects/8fc5873505c64410a64175e5408f9093` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:06.927873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fc5873505c64410a64175e5408f9093)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:06.929803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:06.977897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fc5873505c64410a64175e5408f9093', 'request_id': 'req-7e7b57e5-87f2-449e-866a-7e03bdbc888b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.986262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fc5873505c64410a64175e5408f9093', 'request_id': 'req-7e7b57e5-87f2-449e-866a-7e03bdbc888b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:06.992006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fc5873505c64410a64175e5408f9093', 'request_id': 'req-7e7b57e5-87f2-449e-866a-7e03bdbc888b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:07.005081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] The token cache is being invalidate because project 8fc5873505c64410a64175e5408f9093 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:07.005395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:07.006577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7e7b57e5-87f2-449e-866a-7e03bdbc888b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:07.008352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1107/2227] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:06 2022] DELETE /identity/v3/projects/8fc5873505c64410a64175e5408f9093 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:07.014214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c5d9-5fd7-45f1-bb45-de8375c3734d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.014588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c5d9-5fd7-45f1-bb45-de8375c3734d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.015025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c5d9-5fd7-45f1-bb45-de8375c3734d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.037374 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7431c5d9-5fd7-45f1-bb45-de8375c3734d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:07.038587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7431c5d9-5fd7-45f1-bb45-de8375c3734d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:07.102696 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1121/2228] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:07.112038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6ab515-c08e-4311-9582-bc07bd2361d9 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.112323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6ab515-c08e-4311-9582-bc07bd2361d9 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.112596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6ab515-c08e-4311-9582-bc07bd2361d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.132383 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2e6ab515-c08e-4311-9582-bc07bd2361d9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:07.133561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2e6ab515-c08e-4311-9582-bc07bd2361d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:07.141890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1108/2229] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:07.146197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7431c5d9-5fd7-45f1-bb45-de8375c3734d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.156075 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7431c5d9-5fd7-45f1-bb45-de8375c3734d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.160733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.161331 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.161609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.161884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.162532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:12:07.162837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.164492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.177073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1122/2230] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:12:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:12:07.181031 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6ab515-c08e-4311-9582-bc07bd2361d9 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.191210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6ab515-c08e-4311-9582-bc07bd2361d9 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.195964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9579d2-6acf-48d0-8d3d-3c64e772c1d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.196559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9579d2-6acf-48d0-8d3d-3c64e772c1d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.196839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9579d2-6acf-48d0-8d3d-3c64e772c1d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.197112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9579d2-6acf-48d0-8d3d-3c64e772c1d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.197774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9579d2-6acf-48d0-8d3d-3c64e772c1d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.199656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9579d2-6acf-48d0-8d3d-3c64e772c1d9 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.227797 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1109/2231] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:07 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:12:07.232060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.243917 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdb0eff-cd5e-4a16-ab71-1f96dbafd76b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.304991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b4481b-cc16-4162-aaf5-2829f353399a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.305602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4481b-cc16-4162-aaf5-2829f353399a admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.305881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4481b-cc16-4162-aaf5-2829f353399a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.306153 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4481b-cc16-4162-aaf5-2829f353399a admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.306786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b4481b-cc16-4162-aaf5-2829f353399a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.308789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b4481b-cc16-4162-aaf5-2829f353399a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.333606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1123/2232] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:07 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:07.337845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9579d2-6acf-48d0-8d3d-3c64e772c1d9 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.343235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:07.349752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9579d2-6acf-48d0-8d3d-3c64e772c1d9 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.355353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc1d580-fd20-4af1-8036-1880a7ce358a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.355991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc1d580-fd20-4af1-8036-1880a7ce358a admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.356271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc1d580-fd20-4af1-8036-1880a7ce358a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.356539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc1d580-fd20-4af1-8036-1880a7ce358a admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.357465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc1d580-fd20-4af1-8036-1880a7ce358a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.359052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc1d580-fd20-4af1-8036-1880a7ce358a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.369516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1110/2233] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:07.373728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b4481b-cc16-4162-aaf5-2829f353399a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.385558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b4481b-cc16-4162-aaf5-2829f353399a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.390086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b1d46a-488e-4f38-9d4f-d359bfb31de4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.390685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b1d46a-488e-4f38-9d4f-d359bfb31de4 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.391564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b1d46a-488e-4f38-9d4f-d359bfb31de4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.391874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b1d46a-488e-4f38-9d4f-d359bfb31de4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.392637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b1d46a-488e-4f38-9d4f-d359bfb31de4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.394229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b1d46a-488e-4f38-9d4f-d359bfb31de4 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.405252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1124/2234] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:07.410000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc1d580-fd20-4af1-8036-1880a7ce358a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.419942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc1d580-fd20-4af1-8036-1880a7ce358a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.424719 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905f5761-77b9-4abd-9467-4c8081004a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.425365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f5761-77b9-4abd-9467-4c8081004a95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.425641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f5761-77b9-4abd-9467-4c8081004a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.425914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f5761-77b9-4abd-9467-4c8081004a95 admin admin] PATH_INFO: `/v3/projects/5ab14b34bd9040ce9519c7be0f9e376a/users/647cd9933e85449fa47aca92e4161e83/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.443190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905f5761-77b9-4abd-9467-4c8081004a95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ab14b34bd9040ce9519c7be0f9e376a, user_id=647cd9933e85449fa47aca92e4161e83, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.445789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905f5761-77b9-4abd-9467-4c8081004a95 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.461662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1111/2235] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:07 2022] PUT /identity/v3/projects/5ab14b34bd9040ce9519c7be0f9e376a/users/647cd9933e85449fa47aca92e4161e83/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:07.472808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb24f3-0711-4bc3-aa9d-af5786d7b051 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.473285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb24f3-0711-4bc3-aa9d-af5786d7b051 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.473390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb24f3-0711-4bc3-aa9d-af5786d7b051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.481327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:07.505207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-abcb24f3-0711-4bc3-aa9d-af5786d7b051 None None] MFA Rules not processed for user `647cd9933e85449fa47aca92e4161e83`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:07.507063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-abcb24f3-0711-4bc3-aa9d-af5786d7b051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:07.573835 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1125/2236] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:07.578988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905f5761-77b9-4abd-9467-4c8081004a95 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.642934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905f5761-77b9-4abd-9467-4c8081004a95 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.647549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ad7f14-17af-44c5-8864-13921a622c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.648191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ad7f14-17af-44c5-8864-13921a622c81 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.648474 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ad7f14-17af-44c5-8864-13921a622c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.648774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ad7f14-17af-44c5-8864-13921a622c81 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.649408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ad7f14-17af-44c5-8864-13921a622c81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.651302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ad7f14-17af-44c5-8864-13921a622c81 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.678801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1112/2237] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:07 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:07.683080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb24f3-0711-4bc3-aa9d-af5786d7b051 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.696722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb24f3-0711-4bc3-aa9d-af5786d7b051 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.758333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cf1647-5040-4ba9-98ef-9cf52a88ba9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.758954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cf1647-5040-4ba9-98ef-9cf52a88ba9d admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.759235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cf1647-5040-4ba9-98ef-9cf52a88ba9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.759509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cf1647-5040-4ba9-98ef-9cf52a88ba9d admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.760190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cf1647-5040-4ba9-98ef-9cf52a88ba9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.762224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cf1647-5040-4ba9-98ef-9cf52a88ba9d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.786716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1126/2238] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:07 2022] POST /identity/v3/users => generated 299 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:07.791445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ad7f14-17af-44c5-8864-13921a622c81 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.802442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ad7f14-17af-44c5-8864-13921a622c81 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.807437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e314fda1-1254-4372-a6dc-961e27bbe9e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.808063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e314fda1-1254-4372-a6dc-961e27bbe9e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.808341 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e314fda1-1254-4372-a6dc-961e27bbe9e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.808611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e314fda1-1254-4372-a6dc-961e27bbe9e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.809520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e314fda1-1254-4372-a6dc-961e27bbe9e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.811097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e314fda1-1254-4372-a6dc-961e27bbe9e2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.825032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1113/2239] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:07.827426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cf1647-5040-4ba9-98ef-9cf52a88ba9d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.838199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cf1647-5040-4ba9-98ef-9cf52a88ba9d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.842982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807fd0f3-c110-4e38-a9cf-ae727a9e5a51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.843639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807fd0f3-c110-4e38-a9cf-ae727a9e5a51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.844018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807fd0f3-c110-4e38-a9cf-ae727a9e5a51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.844809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807fd0f3-c110-4e38-a9cf-ae727a9e5a51 admin admin] PATH_INFO: `/v3/projects/aa709a15b5e54023ae60f5ecdef2d049/users/12b6aa9e8f54435f947ac1d437ef4a0c/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.861617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807fd0f3-c110-4e38-a9cf-ae727a9e5a51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa709a15b5e54023ae60f5ecdef2d049, user_id=12b6aa9e8f54435f947ac1d437ef4a0c, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.864687 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807fd0f3-c110-4e38-a9cf-ae727a9e5a51 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.879354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1127/2240] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:07 2022] PUT /identity/v3/projects/aa709a15b5e54023ae60f5ecdef2d049/users/12b6aa9e8f54435f947ac1d437ef4a0c/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:07.883456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e314fda1-1254-4372-a6dc-961e27bbe9e2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.894984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e314fda1-1254-4372-a6dc-961e27bbe9e2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.957092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c739ff78-fdf8-4123-99e5-701738e23905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.957697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c739ff78-fdf8-4123-99e5-701738e23905 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.957976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c739ff78-fdf8-4123-99e5-701738e23905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.958247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c739ff78-fdf8-4123-99e5-701738e23905 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.958928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c739ff78-fdf8-4123-99e5-701738e23905 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.960553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c739ff78-fdf8-4123-99e5-701738e23905 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:07.970137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1114/2241] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:07 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:07.974301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807fd0f3-c110-4e38-a9cf-ae727a9e5a51 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:07.984452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807fd0f3-c110-4e38-a9cf-ae727a9e5a51 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:07.989804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:07.990442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:07.990719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:07.990990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] PATH_INFO: `/v3/projects/aa709a15b5e54023ae60f5ecdef2d049/users/12b6aa9e8f54435f947ac1d437ef4a0c/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:07.992947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa709a15b5e54023ae60f5ecdef2d049, user_id=12b6aa9e8f54435f947ac1d437ef4a0c, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:07.995578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:08.007267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1128/2242] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:07 2022] PUT /identity/v3/projects/aa709a15b5e54023ae60f5ecdef2d049/users/12b6aa9e8f54435f947ac1d437ef4a0c/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:08.017888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46790d21-463a-4da1-b584-15893184a307 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:08.018189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46790d21-463a-4da1-b584-15893184a307 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:08.018453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46790d21-463a-4da1-b584-15893184a307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:08.046434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-46790d21-463a-4da1-b584-15893184a307 None None] MFA Rules not processed for user `12b6aa9e8f54435f947ac1d437ef4a0c`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:08.048373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-46790d21-463a-4da1-b584-15893184a307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:08.073764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:08.116368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1115/2243] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:08 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:08.127315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:08.129220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:08.182600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:08.189188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f041622-7d5b-41bf-813f-1b402911b974 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:08.193114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:08.193713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:08.193995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:08.194297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:08.253310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:08.254691 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:08.264112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1129/2244] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:15.694633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46790d21-463a-4da1-b584-15893184a307 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:15.696051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-46790d21-463a-4da1-b584-15893184a307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:15.707702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46790d21-463a-4da1-b584-15893184a307 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:15.711193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8d63e1-752d-4f19-866f-b14360468aea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:15.711822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8d63e1-752d-4f19-866f-b14360468aea service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:15.712103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8d63e1-752d-4f19-866f-b14360468aea service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:15.712375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8d63e1-752d-4f19-866f-b14360468aea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:15.721671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8d63e1-752d-4f19-866f-b14360468aea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:15.723068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8d63e1-752d-4f19-866f-b14360468aea service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:15.732678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1116/2245] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:15 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:16.041521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:16.042987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:16.122271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb5dba-1f3b-4002-b83a-71bcc3f581f4 service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:16.130137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:16.130777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:16.131058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:16.131333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] PATH_INFO: `/v3/users/6ceffbdb03954009bb4b6dd22aba7ac0` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:16.132491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ceffbdb03954009bb4b6dd22aba7ac0)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:16.134450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:16.186782 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ceffbdb03954009bb4b6dd22aba7ac0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.194000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ceffbdb03954009bb4b6dd22aba7ac0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.194619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ceffbdb03954009bb4b6dd22aba7ac0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.195234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ceffbdb03954009bb4b6dd22aba7ac0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.202214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ceffbdb03954009bb4b6dd22aba7ac0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.220136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1130/2246] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:16 2022] DELETE /identity/v3/users/6ceffbdb03954009bb4b6dd22aba7ac0 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:16.225351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54991cf0-08bb-44ad-b958-c579dab2e45d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:16.225640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54991cf0-08bb-44ad-b958-c579dab2e45d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:16.225920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54991cf0-08bb-44ad-b958-c579dab2e45d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:16.245458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-54991cf0-08bb-44ad-b958-c579dab2e45d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:16.246676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-54991cf0-08bb-44ad-b958-c579dab2e45d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:16.304997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1117/2247] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:16.315668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:16.317371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:16.339612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:16.378205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898cfeac-d73e-4467-bb7b-20f777284274 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:16.381816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d63fbdd-731c-442b-926c-1ebd172a33fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:16.382415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d63fbdd-731c-442b-926c-1ebd172a33fb service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:16.382692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d63fbdd-731c-442b-926c-1ebd172a33fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:16.382988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d63fbdd-731c-442b-926c-1ebd172a33fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:16.391908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d63fbdd-731c-442b-926c-1ebd172a33fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:16.393354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d63fbdd-731c-442b-926c-1ebd172a33fb service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:16.402860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1131/2248] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:16.696048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54991cf0-08bb-44ad-b958-c579dab2e45d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:16.698169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-54991cf0-08bb-44ad-b958-c579dab2e45d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:16.715024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54991cf0-08bb-44ad-b958-c579dab2e45d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:16.725760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:16.726762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:16.727264 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:16.727789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] PATH_INFO: `/v3/projects/2762041c21af407c82eeb40130ea00f0` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:16.729950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] RBAC: Authorizing `identity:delete_project(project_id=2762041c21af407c82eeb40130ea00f0)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:16.731983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:16.782351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2762041c21af407c82eeb40130ea00f0', 'request_id': 'req-11e51a80-39fb-47fc-b668-a7248b2b385d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.794663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2762041c21af407c82eeb40130ea00f0', 'request_id': 'req-11e51a80-39fb-47fc-b668-a7248b2b385d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.801087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2762041c21af407c82eeb40130ea00f0', 'request_id': 'req-11e51a80-39fb-47fc-b668-a7248b2b385d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.813522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] The token cache is being invalidate because project 2762041c21af407c82eeb40130ea00f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:16.813824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.814673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-11e51a80-39fb-47fc-b668-a7248b2b385d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:16.816646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1118/2249] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:16 2022] DELETE /identity/v3/projects/2762041c21af407c82eeb40130ea00f0 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:16.823190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47cc448-4168-4671-ad9c-898294ef7d51 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:16.823794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47cc448-4168-4671-ad9c-898294ef7d51 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:16.824283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47cc448-4168-4671-ad9c-898294ef7d51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:16.844777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-b47cc448-4168-4671-ad9c-898294ef7d51 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:16.846352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b47cc448-4168-4671-ad9c-898294ef7d51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:16.914199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1132/2250] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:16.928678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1672b88-cdf8-40ea-9b70-20f351a5c43a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:16.929080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1672b88-cdf8-40ea-9b70-20f351a5c43a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:16.929361 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1672b88-cdf8-40ea-9b70-20f351a5c43a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:16.950419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b1672b88-cdf8-40ea-9b70-20f351a5c43a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:16.951635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b1672b88-cdf8-40ea-9b70-20f351a5c43a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:16.963544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1119/2251] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:12:16.965414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47cc448-4168-4671-ad9c-898294ef7d51 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:16.976723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47cc448-4168-4671-ad9c-898294ef7d51 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:16.982314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:16.982922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:16.983199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:16.983473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:16.984255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:12:16.985111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:16.986666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:17.001034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1133/2252] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:12:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:12:17.016331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1672b88-cdf8-40ea-9b70-20f351a5c43a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:17.025727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1672b88-cdf8-40ea-9b70-20f351a5c43a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:17.030893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e992a3c-2afc-49f6-99b7-7c59a7d9ac46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:17.031522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e992a3c-2afc-49f6-99b7-7c59a7d9ac46 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:17.031820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e992a3c-2afc-49f6-99b7-7c59a7d9ac46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:17.032156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e992a3c-2afc-49f6-99b7-7c59a7d9ac46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:17.032727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e992a3c-2afc-49f6-99b7-7c59a7d9ac46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:17.035624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e992a3c-2afc-49f6-99b7-7c59a7d9ac46 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:17.064029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1120/2253] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:17 2022] POST /identity/v3/projects => generated 363 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:17.069808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:17.081677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341a4ed6-1ffc-427c-b87f-b93025cff5e1 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:17.144496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1442cf9d-3b64-4205-9e4c-0a6fca42365e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:17.145944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1442cf9d-3b64-4205-9e4c-0a6fca42365e admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:17.146123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1442cf9d-3b64-4205-9e4c-0a6fca42365e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:17.146514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1442cf9d-3b64-4205-9e4c-0a6fca42365e admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:17.147389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1442cf9d-3b64-4205-9e4c-0a6fca42365e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:17.149464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1442cf9d-3b64-4205-9e4c-0a6fca42365e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:17.178121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1134/2254] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:17 2022] POST /identity/v3/users => generated 282 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:17.182822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e992a3c-2afc-49f6-99b7-7c59a7d9ac46 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:17.197980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:17.202236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e992a3c-2afc-49f6-99b7-7c59a7d9ac46 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:17.208980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9fe87-b4ca-4d50-981e-d794b6c345e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:17.209698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9fe87-b4ca-4d50-981e-d794b6c345e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:17.210043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9fe87-b4ca-4d50-981e-d794b6c345e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:17.210484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9fe87-b4ca-4d50-981e-d794b6c345e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:17.211123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9fe87-b4ca-4d50-981e-d794b6c345e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:17.213212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9fe87-b4ca-4d50-981e-d794b6c345e2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:17.229360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1121/2255] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:17.233504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1442cf9d-3b64-4205-9e4c-0a6fca42365e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:17.244898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1442cf9d-3b64-4205-9e4c-0a6fca42365e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:17.255310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0932eb4-982b-4dd6-9f08-c2d5bce5e367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:17.256000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0932eb4-982b-4dd6-9f08-c2d5bce5e367 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:17.256290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0932eb4-982b-4dd6-9f08-c2d5bce5e367 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:17.257443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0932eb4-982b-4dd6-9f08-c2d5bce5e367 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:17.258125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0932eb4-982b-4dd6-9f08-c2d5bce5e367 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:17.259783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0932eb4-982b-4dd6-9f08-c2d5bce5e367 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:17.271763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1135/2256] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:17.275417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9fe87-b4ca-4d50-981e-d794b6c345e2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:17.287860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9fe87-b4ca-4d50-981e-d794b6c345e2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:17.293891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:17.294671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:17.294973 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:17.295288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] PATH_INFO: `/v3/projects/19ce5c35248e4e58833cc3b7cfa703d2/users/434a2a9c6c6a473080a9bc84dfcc1e1e/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:17.325188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=19ce5c35248e4e58833cc3b7cfa703d2, user_id=434a2a9c6c6a473080a9bc84dfcc1e1e, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:17.328354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:17.344191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1122/2257] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:17 2022] PUT /identity/v3/projects/19ce5c35248e4e58833cc3b7cfa703d2/users/434a2a9c6c6a473080a9bc84dfcc1e1e/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:17.358406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b132fb-4ddb-401f-b0ed-9c9118bd1bd3 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:17.358406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b132fb-4ddb-401f-b0ed-9c9118bd1bd3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:17.358406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b132fb-4ddb-401f-b0ed-9c9118bd1bd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:17.392146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-82b132fb-4ddb-401f-b0ed-9c9118bd1bd3 None None] MFA Rules not processed for user `434a2a9c6c6a473080a9bc84dfcc1e1e`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:17.394477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-82b132fb-4ddb-401f-b0ed-9c9118bd1bd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:17.412880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:17.464977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1136/2258] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:17 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:17.476999 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:17.478648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:17.535715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c3f49-16c0-456a-950a-9198e2fdd2dc admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:17.539434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49bed1e-cf24-48ac-90d4-01e0a0febe74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:17.540147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49bed1e-cf24-48ac-90d4-01e0a0febe74 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:17.540432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49bed1e-cf24-48ac-90d4-01e0a0febe74 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:17.540722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49bed1e-cf24-48ac-90d4-01e0a0febe74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:17.549128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49bed1e-cf24-48ac-90d4-01e0a0febe74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:17.550744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49bed1e-cf24-48ac-90d4-01e0a0febe74 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:17.559780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1123/2259] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:17 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:20.718805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b132fb-4ddb-401f-b0ed-9c9118bd1bd3 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:20.720345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-82b132fb-4ddb-401f-b0ed-9c9118bd1bd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:20.788218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b132fb-4ddb-401f-b0ed-9c9118bd1bd3 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:20.792921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:20.793561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:20.793847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:20.794121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] PATH_INFO: `/v3/users/647cd9933e85449fa47aca92e4161e83` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:20.795237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=647cd9933e85449fa47aca92e4161e83)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:20.797290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:20.843116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '647cd9933e85449fa47aca92e4161e83'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:20.850731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '647cd9933e85449fa47aca92e4161e83'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:20.851379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '647cd9933e85449fa47aca92e4161e83'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:20.852040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '647cd9933e85449fa47aca92e4161e83'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:20.859020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '647cd9933e85449fa47aca92e4161e83'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:20.876663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1137/2260] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:20 2022] DELETE /identity/v3/users/647cd9933e85449fa47aca92e4161e83 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:20.881218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df32564-7eb6-408b-a1ee-8dbc78408f21 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:20.881688 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df32564-7eb6-408b-a1ee-8dbc78408f21 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:20.881779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df32564-7eb6-408b-a1ee-8dbc78408f21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:20.900274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-0df32564-7eb6-408b-a1ee-8dbc78408f21 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:20.901466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0df32564-7eb6-408b-a1ee-8dbc78408f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:20.959258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1124/2261] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:20.968914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:20.970366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:21.015392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:21.032372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d38260-6719-40a7-ad30-a1dec78bf4a2 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:21.036625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1ea5e4-1b6c-4db5-ac7b-33e89a6273d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:21.037253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1ea5e4-1b6c-4db5-ac7b-33e89a6273d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:21.037537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1ea5e4-1b6c-4db5-ac7b-33e89a6273d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:21.037811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1ea5e4-1b6c-4db5-ac7b-33e89a6273d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:21.046719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1ea5e4-1b6c-4db5-ac7b-33e89a6273d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:21.048182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1ea5e4-1b6c-4db5-ac7b-33e89a6273d4 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:21.058108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1138/2262] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:21.337691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df32564-7eb6-408b-a1ee-8dbc78408f21 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:21.339219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0df32564-7eb6-408b-a1ee-8dbc78408f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:21.351720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df32564-7eb6-408b-a1ee-8dbc78408f21 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:21.356330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:21.356954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:21.357230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:21.357509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] PATH_INFO: `/v3/projects/5ab14b34bd9040ce9519c7be0f9e376a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:21.358688 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ab14b34bd9040ce9519c7be0f9e376a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:21.360693 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:21.408748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ab14b34bd9040ce9519c7be0f9e376a', 'request_id': 'req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.416323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ab14b34bd9040ce9519c7be0f9e376a', 'request_id': 'req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.422057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ab14b34bd9040ce9519c7be0f9e376a', 'request_id': 'req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.424490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:21.436746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] The token cache is being invalidate because project 5ab14b34bd9040ce9519c7be0f9e376a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:21.437035 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.437680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.439147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1125/2263] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:21 2022] DELETE /identity/v3/projects/5ab14b34bd9040ce9519c7be0f9e376a => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:21.443795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1ea5e4-1b6c-4db5-ac7b-33e89a6273d4 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:21.453204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1ea5e4-1b6c-4db5-ac7b-33e89a6273d4 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:21.513292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:21.513962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:21.514242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:21.514601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] PATH_INFO: `/v3/users/12b6aa9e8f54435f947ac1d437ef4a0c` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:21.515777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] RBAC: Authorizing `identity:delete_user(user_id=12b6aa9e8f54435f947ac1d437ef4a0c)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:21.517845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:21.570714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12b6aa9e8f54435f947ac1d437ef4a0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.577973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12b6aa9e8f54435f947ac1d437ef4a0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.578633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12b6aa9e8f54435f947ac1d437ef4a0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.579262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12b6aa9e8f54435f947ac1d437ef4a0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.587608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e2be8e4c-abea-44a9-8ad1-15773684105a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12b6aa9e8f54435f947ac1d437ef4a0c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:21.623383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1139/2264] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:21 2022] DELETE /identity/v3/users/12b6aa9e8f54435f947ac1d437ef4a0c => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:21.919052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:21.920580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:21.934291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f3e9f-8cc9-4232-bcf5-608b8fae2a24 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:21.998805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:21.999499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:21.999836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.000080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] PATH_INFO: `/v3/projects/aa709a15b5e54023ae60f5ecdef2d049` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.001202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa709a15b5e54023ae60f5ecdef2d049)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.003946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.050238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa709a15b5e54023ae60f5ecdef2d049', 'request_id': 'req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:22.057092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa709a15b5e54023ae60f5ecdef2d049', 'request_id': 'req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:22.063250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa709a15b5e54023ae60f5ecdef2d049', 'request_id': 'req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:22.075636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] The token cache is being invalidate because project aa709a15b5e54023ae60f5ecdef2d049 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:22.075958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:22.076621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-01ea2a5c-f76c-40bc-94d0-8b1c9c85443e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:22.078088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1126/2265] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:21 2022] DELETE /identity/v3/projects/aa709a15b5e54023ae60f5ecdef2d049 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:12:22.084509 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e151e0dc-393e-468e-b49f-49d1dc1c1609 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.085066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e151e0dc-393e-468e-b49f-49d1dc1c1609 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.085066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e151e0dc-393e-468e-b49f-49d1dc1c1609 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.104753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e151e0dc-393e-468e-b49f-49d1dc1c1609 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:22.105946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e151e0dc-393e-468e-b49f-49d1dc1c1609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:22.156220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:22.165558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1140/2266] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:22.175532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fe371f-4532-4fab-92d4-f392f17fbcdf None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.175856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fe371f-4532-4fab-92d4-f392f17fbcdf None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.176133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fe371f-4532-4fab-92d4-f392f17fbcdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.197299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f8fe371f-4532-4fab-92d4-f392f17fbcdf None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:22.198503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f8fe371f-4532-4fab-92d4-f392f17fbcdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:22.206803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1127/2267] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:22.211024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e151e0dc-393e-468e-b49f-49d1dc1c1609 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.220597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e151e0dc-393e-468e-b49f-49d1dc1c1609 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.225751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.226346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.226626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.226900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.227546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:12:22.227887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.229489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.241748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1141/2268] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:12:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:12:22.245967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fe371f-4532-4fab-92d4-f392f17fbcdf None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.255311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fe371f-4532-4fab-92d4-f392f17fbcdf None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.259922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5549a114-85d0-4ea3-b9fe-1dc7d56c1481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.260516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5549a114-85d0-4ea3-b9fe-1dc7d56c1481 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.260798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5549a114-85d0-4ea3-b9fe-1dc7d56c1481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.261074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5549a114-85d0-4ea3-b9fe-1dc7d56c1481 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.261705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5549a114-85d0-4ea3-b9fe-1dc7d56c1481 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.263934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5549a114-85d0-4ea3-b9fe-1dc7d56c1481 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.295015 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1128/2269] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:22 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:22.296174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.307634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ffeac-5c7c-4cf2-b714-f2948d88e571 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.370254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbf94ea-8a7e-479f-be66-cb2d21b306aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.370854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbf94ea-8a7e-479f-be66-cb2d21b306aa admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.371133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbf94ea-8a7e-479f-be66-cb2d21b306aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.371407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbf94ea-8a7e-479f-be66-cb2d21b306aa admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.372070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbf94ea-8a7e-479f-be66-cb2d21b306aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.373989 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbf94ea-8a7e-479f-be66-cb2d21b306aa admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.394586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1142/2270] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:22 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:22.398618 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5549a114-85d0-4ea3-b9fe-1dc7d56c1481 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.410293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5549a114-85d0-4ea3-b9fe-1dc7d56c1481 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.414772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf53ed4-2526-40c5-b267-845c3dc99521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.415359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf53ed4-2526-40c5-b267-845c3dc99521 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.415637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf53ed4-2526-40c5-b267-845c3dc99521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.415939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf53ed4-2526-40c5-b267-845c3dc99521 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.416607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf53ed4-2526-40c5-b267-845c3dc99521 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.418181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf53ed4-2526-40c5-b267-845c3dc99521 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.429386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1129/2271] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:22.433315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbf94ea-8a7e-479f-be66-cb2d21b306aa admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.443396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbf94ea-8a7e-479f-be66-cb2d21b306aa admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.448027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520fca5f-a5aa-4bcf-a09e-9a50ab0be0fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.448616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520fca5f-a5aa-4bcf-a09e-9a50ab0be0fa admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.448895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520fca5f-a5aa-4bcf-a09e-9a50ab0be0fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.449165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520fca5f-a5aa-4bcf-a09e-9a50ab0be0fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.449830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520fca5f-a5aa-4bcf-a09e-9a50ab0be0fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.451458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520fca5f-a5aa-4bcf-a09e-9a50ab0be0fa admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.461870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1143/2272] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:22.465873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf53ed4-2526-40c5-b267-845c3dc99521 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.476159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf53ed4-2526-40c5-b267-845c3dc99521 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.480816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8d88ea-70b2-4131-8ad3-bf02cc4d9e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.481447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d88ea-70b2-4131-8ad3-bf02cc4d9e13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.481722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d88ea-70b2-4131-8ad3-bf02cc4d9e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.481991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d88ea-70b2-4131-8ad3-bf02cc4d9e13 admin admin] PATH_INFO: `/v3/projects/bf5438fe888f47e6b2b321184d444a64/users/e872c2adf0834756b48faec2d44aa116/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.496731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8d88ea-70b2-4131-8ad3-bf02cc4d9e13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf5438fe888f47e6b2b321184d444a64, user_id=e872c2adf0834756b48faec2d44aa116, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.499347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8d88ea-70b2-4131-8ad3-bf02cc4d9e13 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.512037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1130/2273] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:22 2022] PUT /identity/v3/projects/bf5438fe888f47e6b2b321184d444a64/users/e872c2adf0834756b48faec2d44aa116/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:22.523556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ea198d-c7bb-4a9f-bf56-4ad5ca955e95 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.523867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ea198d-c7bb-4a9f-bf56-4ad5ca955e95 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.524144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ea198d-c7bb-4a9f-bf56-4ad5ca955e95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.552789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-85ea198d-c7bb-4a9f-bf56-4ad5ca955e95 None None] MFA Rules not processed for user `e872c2adf0834756b48faec2d44aa116`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:22.553991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-85ea198d-c7bb-4a9f-bf56-4ad5ca955e95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:22.617788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1144/2274] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:12:22.622247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8d88ea-70b2-4131-8ad3-bf02cc4d9e13 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.647667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:22.692152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8d88ea-70b2-4131-8ad3-bf02cc4d9e13 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.696918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a760e84-d0b4-4753-8a97-f0d0075342fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.698700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a760e84-d0b4-4753-8a97-f0d0075342fd admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.698985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a760e84-d0b4-4753-8a97-f0d0075342fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.699263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a760e84-d0b4-4753-8a97-f0d0075342fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.699948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a760e84-d0b4-4753-8a97-f0d0075342fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.701848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a760e84-d0b4-4753-8a97-f0d0075342fd admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.733040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1131/2275] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:22 2022] POST /identity/v3/projects => generated 383 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:22.734652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ea198d-c7bb-4a9f-bf56-4ad5ca955e95 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.746313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ea198d-c7bb-4a9f-bf56-4ad5ca955e95 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.805658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:22.815114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa2ee4c-74b1-4a80-8efd-702bb9009225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.815713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa2ee4c-74b1-4a80-8efd-702bb9009225 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.816046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa2ee4c-74b1-4a80-8efd-702bb9009225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.816324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa2ee4c-74b1-4a80-8efd-702bb9009225 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.817070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa2ee4c-74b1-4a80-8efd-702bb9009225 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.819188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa2ee4c-74b1-4a80-8efd-702bb9009225 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.844895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1145/2276] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:22 2022] POST /identity/v3/users => generated 292 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:22.849171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a760e84-d0b4-4753-8a97-f0d0075342fd admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.864659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a760e84-d0b4-4753-8a97-f0d0075342fd admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.869506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7bc37c-7fb0-4d7e-a6fb-ac9cd9ef1432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.870305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7bc37c-7fb0-4d7e-a6fb-ac9cd9ef1432 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.870772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7bc37c-7fb0-4d7e-a6fb-ac9cd9ef1432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.870993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7bc37c-7fb0-4d7e-a6fb-ac9cd9ef1432 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.871717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7bc37c-7fb0-4d7e-a6fb-ac9cd9ef1432 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.873451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7bc37c-7fb0-4d7e-a6fb-ac9cd9ef1432 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.885221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1132/2277] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:22.889477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa2ee4c-74b1-4a80-8efd-702bb9009225 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.900221 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa2ee4c-74b1-4a80-8efd-702bb9009225 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.904960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039cb11-be33-49a4-aa85-c14beea2cfc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.905554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039cb11-be33-49a4-aa85-c14beea2cfc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.905830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039cb11-be33-49a4-aa85-c14beea2cfc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.906107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039cb11-be33-49a4-aa85-c14beea2cfc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.906845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039cb11-be33-49a4-aa85-c14beea2cfc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.909295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039cb11-be33-49a4-aa85-c14beea2cfc9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.919425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1146/2278] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:22.923575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7bc37c-7fb0-4d7e-a6fb-ac9cd9ef1432 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:22.933608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7bc37c-7fb0-4d7e-a6fb-ac9cd9ef1432 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:22.938700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0532caa-b598-48a4-a85b-d5987f6ea095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:22.939454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0532caa-b598-48a4-a85b-d5987f6ea095 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.939867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0532caa-b598-48a4-a85b-d5987f6ea095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.940241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0532caa-b598-48a4-a85b-d5987f6ea095 admin admin] PATH_INFO: `/v3/projects/1ef4240bcd94410aaba6024aa171aa8d/users/f77ce9e4eff3416aa5d3c130c02c073a/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:22.956097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0532caa-b598-48a4-a85b-d5987f6ea095 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ef4240bcd94410aaba6024aa171aa8d, user_id=f77ce9e4eff3416aa5d3c130c02c073a, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:22.958739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0532caa-b598-48a4-a85b-d5987f6ea095 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:22.972433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1133/2279] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:22 2022] PUT /identity/v3/projects/1ef4240bcd94410aaba6024aa171aa8d/users/f77ce9e4eff3416aa5d3c130c02c073a/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:22.983453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78285ece-0e7b-4a78-af10-ec642f2eb581 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:22.983777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78285ece-0e7b-4a78-af10-ec642f2eb581 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:22.984062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78285ece-0e7b-4a78-af10-ec642f2eb581 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.012797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-78285ece-0e7b-4a78-af10-ec642f2eb581 None None] MFA Rules not processed for user `f77ce9e4eff3416aa5d3c130c02c073a`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:23.013932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-78285ece-0e7b-4a78-af10-ec642f2eb581 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:23.076908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1147/2280] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:23.081979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0532caa-b598-48a4-a85b-d5987f6ea095 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:23.143170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0532caa-b598-48a4-a85b-d5987f6ea095 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:23.148131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed42b7d-9496-4124-b194-0ce5dcd57193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:23.148762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed42b7d-9496-4124-b194-0ce5dcd57193 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:23.149041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed42b7d-9496-4124-b194-0ce5dcd57193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:23.149374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed42b7d-9496-4124-b194-0ce5dcd57193 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.150013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed42b7d-9496-4124-b194-0ce5dcd57193 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:23.151945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed42b7d-9496-4124-b194-0ce5dcd57193 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:23.177628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1134/2281] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:23 2022] POST /identity/v3/projects => generated 385 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:23.181938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78285ece-0e7b-4a78-af10-ec642f2eb581 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:23.194156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78285ece-0e7b-4a78-af10-ec642f2eb581 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:23.257831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcfcd9c-d70b-4ad9-b9d0-f16176022caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:23.258437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcfcd9c-d70b-4ad9-b9d0-f16176022caf admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:23.258713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcfcd9c-d70b-4ad9-b9d0-f16176022caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:23.258987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcfcd9c-d70b-4ad9-b9d0-f16176022caf admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.259612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcfcd9c-d70b-4ad9-b9d0-f16176022caf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:23.261552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcfcd9c-d70b-4ad9-b9d0-f16176022caf admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:23.282101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1148/2282] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:23 2022] POST /identity/v3/users => generated 299 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:23.286162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed42b7d-9496-4124-b194-0ce5dcd57193 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:23.296333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed42b7d-9496-4124-b194-0ce5dcd57193 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:23.301106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a019b-f75b-43a5-84b9-4ceecaa5e2bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:23.301708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a019b-f75b-43a5-84b9-4ceecaa5e2bd admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:23.301989 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a019b-f75b-43a5-84b9-4ceecaa5e2bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:23.302260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7a019b-f75b-43a5-84b9-4ceecaa5e2bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.302915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7a019b-f75b-43a5-84b9-4ceecaa5e2bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:23.304764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7a019b-f75b-43a5-84b9-4ceecaa5e2bd admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:23.316529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1135/2283] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:23.320769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcfcd9c-d70b-4ad9-b9d0-f16176022caf admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:23.332558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcfcd9c-d70b-4ad9-b9d0-f16176022caf admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:23.337631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615db6b5-67c1-44ef-a666-4415bda4747b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:23.338330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615db6b5-67c1-44ef-a666-4415bda4747b admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:23.338656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615db6b5-67c1-44ef-a666-4415bda4747b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:23.338933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615db6b5-67c1-44ef-a666-4415bda4747b admin admin] PATH_INFO: `/v3/projects/f071042d3ed943679e677dee2c81da97/users/7161ef0c294747fda452e6af1ec2aae4/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.354664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615db6b5-67c1-44ef-a666-4415bda4747b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f071042d3ed943679e677dee2c81da97, user_id=7161ef0c294747fda452e6af1ec2aae4, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:23.357444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615db6b5-67c1-44ef-a666-4415bda4747b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:23.370994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1149/2284] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:23 2022] PUT /identity/v3/projects/f071042d3ed943679e677dee2c81da97/users/7161ef0c294747fda452e6af1ec2aae4/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:23.375062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a019b-f75b-43a5-84b9-4ceecaa5e2bd admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:23.385550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7a019b-f75b-43a5-84b9-4ceecaa5e2bd admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:23.428254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:23.442467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcea377-2392-4884-967d-1df246a58655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:23.443069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcea377-2392-4884-967d-1df246a58655 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:23.443353 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcea377-2392-4884-967d-1df246a58655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:23.443628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcea377-2392-4884-967d-1df246a58655 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.444309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcea377-2392-4884-967d-1df246a58655 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:23.445900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcea377-2392-4884-967d-1df246a58655 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:23.455590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1136/2285] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:23 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:23.459932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615db6b5-67c1-44ef-a666-4415bda4747b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:23.470171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615db6b5-67c1-44ef-a666-4415bda4747b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:23.474889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:23.475553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:23.475849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:23.476130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] PATH_INFO: `/v3/projects/f071042d3ed943679e677dee2c81da97/users/7161ef0c294747fda452e6af1ec2aae4/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.478023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f071042d3ed943679e677dee2c81da97, user_id=7161ef0c294747fda452e6af1ec2aae4, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:23.480846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:23.491423 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1150/2286] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:23 2022] PUT /identity/v3/projects/f071042d3ed943679e677dee2c81da97/users/7161ef0c294747fda452e6af1ec2aae4/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:23.501288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58609ea-5296-4567-b89a-e60bdb44b7aa None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:23.501599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58609ea-5296-4567-b89a-e60bdb44b7aa None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:23.501878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58609ea-5296-4567-b89a-e60bdb44b7aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.530155 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d58609ea-5296-4567-b89a-e60bdb44b7aa None None] MFA Rules not processed for user `7161ef0c294747fda452e6af1ec2aae4`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:23.531363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d58609ea-5296-4567-b89a-e60bdb44b7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:23.601031 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1137/2287] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:23 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:23.614501 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:23.616231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:23.636557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:23.676478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ecb18e-5ac6-4fc7-a587-4914d8728b91 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:23.680124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641d9344-9e50-40bb-948f-6879b5af313a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:23.680718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641d9344-9e50-40bb-948f-6879b5af313a service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:23.680991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641d9344-9e50-40bb-948f-6879b5af313a service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:23.681259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641d9344-9e50-40bb-948f-6879b5af313a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:23.736441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641d9344-9e50-40bb-948f-6879b5af313a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:23.738059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641d9344-9e50-40bb-948f-6879b5af313a service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:23.747384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1151/2288] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:26.340123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58609ea-5296-4567-b89a-e60bdb44b7aa None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:26.350608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58609ea-5296-4567-b89a-e60bdb44b7aa None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:26.354145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5bc9a3-fa31-4726-a5f0-4adc120596ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:26.354749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5bc9a3-fa31-4726-a5f0-4adc120596ae service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:26.355059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5bc9a3-fa31-4726-a5f0-4adc120596ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:26.355329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5bc9a3-fa31-4726-a5f0-4adc120596ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:26.416048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5bc9a3-fa31-4726-a5f0-4adc120596ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:26.417487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5bc9a3-fa31-4726-a5f0-4adc120596ae service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:26.426378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1138/2289] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:26.948012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641d9344-9e50-40bb-948f-6879b5af313a service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:26.956740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641d9344-9e50-40bb-948f-6879b5af313a service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:26.960359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3d49e3-4a12-4827-920c-53df95e19c15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:26.960955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d49e3-4a12-4827-920c-53df95e19c15 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:26.961239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d49e3-4a12-4827-920c-53df95e19c15 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:26.961512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3d49e3-4a12-4827-920c-53df95e19c15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:26.970397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3d49e3-4a12-4827-920c-53df95e19c15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:26.972581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3d49e3-4a12-4827-920c-53df95e19c15 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:26.982109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1152/2290] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:26 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:36.106861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5bc9a3-fa31-4726-a5f0-4adc120596ae service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:36.295251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5bc9a3-fa31-4726-a5f0-4adc120596ae service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:36.299740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:36.300370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:36.300652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:36.300922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] PATH_INFO: `/v3/users/e872c2adf0834756b48faec2d44aa116` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:36.302038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e872c2adf0834756b48faec2d44aa116)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:36.304513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:36.310337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:36.350741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e872c2adf0834756b48faec2d44aa116'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:36.357647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e872c2adf0834756b48faec2d44aa116'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:36.358245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e872c2adf0834756b48faec2d44aa116'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:36.358849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e872c2adf0834756b48faec2d44aa116'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:36.365889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e872c2adf0834756b48faec2d44aa116'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:36.383854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1139/2291] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:36 2022] DELETE /identity/v3/users/e872c2adf0834756b48faec2d44aa116 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:36.388448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5591e4ce-5fdc-4a9f-b31d-33766d89a3fd None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:36.388937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5591e4ce-5fdc-4a9f-b31d-33766d89a3fd None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:36.389038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5591e4ce-5fdc-4a9f-b31d-33766d89a3fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:36.408156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5591e4ce-5fdc-4a9f-b31d-33766d89a3fd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:36.409481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5591e4ce-5fdc-4a9f-b31d-33766d89a3fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:36.470552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1153/2292] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:36.481412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:36.482786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:36.540895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed714641-6869-45de-9212-ef4d5d8d71e1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:36.544538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2f6f20-ed01-4703-a0c9-7246d5e9205b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:36.545132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2f6f20-ed01-4703-a0c9-7246d5e9205b service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:36.545405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2f6f20-ed01-4703-a0c9-7246d5e9205b service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:36.545669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2f6f20-ed01-4703-a0c9-7246d5e9205b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:36.555259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2f6f20-ed01-4703-a0c9-7246d5e9205b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:36.556697 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2f6f20-ed01-4703-a0c9-7246d5e9205b service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:36.566054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1140/2293] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:36.919988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5591e4ce-5fdc-4a9f-b31d-33766d89a3fd None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:36.921430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5591e4ce-5fdc-4a9f-b31d-33766d89a3fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:36.934404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5591e4ce-5fdc-4a9f-b31d-33766d89a3fd None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:36.939456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:36.940225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:36.940466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:36.940687 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] PATH_INFO: `/v3/projects/bf5438fe888f47e6b2b321184d444a64` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:36.941815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf5438fe888f47e6b2b321184d444a64)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:36.944145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:37.051886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf5438fe888f47e6b2b321184d444a64', 'request_id': 'req-bc7164c3-2a62-4a19-b83a-208fe491f82d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.056237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:37.064615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf5438fe888f47e6b2b321184d444a64', 'request_id': 'req-bc7164c3-2a62-4a19-b83a-208fe491f82d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.082012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf5438fe888f47e6b2b321184d444a64', 'request_id': 'req-bc7164c3-2a62-4a19-b83a-208fe491f82d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.097029 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] The token cache is being invalidate because project bf5438fe888f47e6b2b321184d444a64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:37.097322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.098061 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.100162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1154/2294] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:36 2022] DELETE /identity/v3/projects/bf5438fe888f47e6b2b321184d444a64 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:37.104687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2f6f20-ed01-4703-a0c9-7246d5e9205b service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:37.115841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2f6f20-ed01-4703-a0c9-7246d5e9205b service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:37.184751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:37.185736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:37.186011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:37.186297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] PATH_INFO: `/v3/users/f77ce9e4eff3416aa5d3c130c02c073a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:37.187610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] RBAC: Authorizing `identity:delete_user(user_id=f77ce9e4eff3416aa5d3c130c02c073a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:37.189539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:37.233092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:37.250377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f77ce9e4eff3416aa5d3c130c02c073a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.259228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f77ce9e4eff3416aa5d3c130c02c073a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.259951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f77ce9e4eff3416aa5d3c130c02c073a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.260565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f77ce9e4eff3416aa5d3c130c02c073a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.272110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f77ce9e4eff3416aa5d3c130c02c073a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.304632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1141/2295] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:37 2022] DELETE /identity/v3/users/f77ce9e4eff3416aa5d3c130c02c073a => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:37.602100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:37.604301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:37.618816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7164c3-2a62-4a19-b83a-208fe491f82d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:37.686444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:37.687086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:37.687373 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:37.687649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] PATH_INFO: `/v3/projects/1ef4240bcd94410aaba6024aa171aa8d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:37.688805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ef4240bcd94410aaba6024aa171aa8d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:37.691399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:37.739808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ef4240bcd94410aaba6024aa171aa8d', 'request_id': 'req-38c2481d-236f-4802-82f6-1baa1f5c48df'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.747135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ef4240bcd94410aaba6024aa171aa8d', 'request_id': 'req-38c2481d-236f-4802-82f6-1baa1f5c48df'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.752885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ef4240bcd94410aaba6024aa171aa8d', 'request_id': 'req-38c2481d-236f-4802-82f6-1baa1f5c48df'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.765705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] The token cache is being invalidate because project 1ef4240bcd94410aaba6024aa171aa8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:37.766001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.767036 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.768602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1155/2296] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:37 2022] DELETE /identity/v3/projects/1ef4240bcd94410aaba6024aa171aa8d => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:12:37.773504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:37.774894 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:37.789305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e138ff9-92e2-43b7-90f5-16ba2c7abece admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:37.849783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:37.850456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:37.850738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:37.851013 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] PATH_INFO: `/v3/users/7161ef0c294747fda452e6af1ec2aae4` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:37.852179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] RBAC: Authorizing `identity:delete_user(user_id=7161ef0c294747fda452e6af1ec2aae4)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:37.854140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:37.903015 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7161ef0c294747fda452e6af1ec2aae4'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.913062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7161ef0c294747fda452e6af1ec2aae4'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.913898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7161ef0c294747fda452e6af1ec2aae4'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.914735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7161ef0c294747fda452e6af1ec2aae4'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.922575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-23859e34-b23d-433d-9982-2eb017046efc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7161ef0c294747fda452e6af1ec2aae4'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:37.945642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1142/2297] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:12:37 2022] DELETE /identity/v3/users/7161ef0c294747fda452e6af1ec2aae4 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:38.222744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:38.224846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:38.242270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c2481d-236f-4802-82f6-1baa1f5c48df admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:38.304748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:38.305402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.305683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.305955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] PATH_INFO: `/v3/projects/f071042d3ed943679e677dee2c81da97` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.307085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f071042d3ed943679e677dee2c81da97)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:38.309117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:38.321546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:38.359267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f071042d3ed943679e677dee2c81da97', 'request_id': 'req-b5e94f27-decb-4409-aa98-4e86aa932ec5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:38.366524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f071042d3ed943679e677dee2c81da97', 'request_id': 'req-b5e94f27-decb-4409-aa98-4e86aa932ec5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:38.372273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f071042d3ed943679e677dee2c81da97', 'request_id': 'req-b5e94f27-decb-4409-aa98-4e86aa932ec5'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:38.383768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] The token cache is being invalidate because project f071042d3ed943679e677dee2c81da97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:12:38.384058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:38.384713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b5e94f27-decb-4409-aa98-4e86aa932ec5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:12:38.386187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1156/2298] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:12:38 2022] DELETE /identity/v3/projects/f071042d3ed943679e677dee2c81da97 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:38.390971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12333ae9-d456-4291-bbf9-95ad3c7b2c8d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.391369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12333ae9-d456-4291-bbf9-95ad3c7b2c8d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.391612 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12333ae9-d456-4291-bbf9-95ad3c7b2c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.412962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-12333ae9-d456-4291-bbf9-95ad3c7b2c8d None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:38.414163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-12333ae9-d456-4291-bbf9-95ad3c7b2c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:38.435852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:38.479906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1143/2299] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:38.487341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96617c56-cbad-4818-bb91-ab821ef2c073 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.487630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96617c56-cbad-4818-bb91-ab821ef2c073 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.487942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96617c56-cbad-4818-bb91-ab821ef2c073 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.509046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-96617c56-cbad-4818-bb91-ab821ef2c073 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:38.510266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-96617c56-cbad-4818-bb91-ab821ef2c073 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:38.518280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1157/2300] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:38.523222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12333ae9-d456-4291-bbf9-95ad3c7b2c8d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:38.533116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12333ae9-d456-4291-bbf9-95ad3c7b2c8d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:38.538523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:38.539125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.539406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.539678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.540364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:12:38.540668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:38.542255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:38.557971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1144/2301] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:12:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:12:38.562807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96617c56-cbad-4818-bb91-ab821ef2c073 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:38.572465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96617c56-cbad-4818-bb91-ab821ef2c073 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:38.577139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a095853c-0dd5-45bb-ab46-174ed78525e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:38.577740 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095853c-0dd5-45bb-ab46-174ed78525e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.578012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095853c-0dd5-45bb-ab46-174ed78525e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.578285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095853c-0dd5-45bb-ab46-174ed78525e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.578916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a095853c-0dd5-45bb-ab46-174ed78525e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:38.580849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a095853c-0dd5-45bb-ab46-174ed78525e2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:38.607884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1158/2302] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:12:38 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:38.612212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:38.623006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc88d90-6024-4122-af3e-d44d9df8c22e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:38.679181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1b0cb8-812a-44bc-962a-0c59621dea33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:38.679810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b0cb8-812a-44bc-962a-0c59621dea33 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.680092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b0cb8-812a-44bc-962a-0c59621dea33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.680360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b0cb8-812a-44bc-962a-0c59621dea33 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.680981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1b0cb8-812a-44bc-962a-0c59621dea33 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:38.682871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1b0cb8-812a-44bc-962a-0c59621dea33 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:38.704278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1145/2303] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:12:38 2022] POST /identity/v3/users => generated 286 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:12:38.708568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a095853c-0dd5-45bb-ab46-174ed78525e2 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:38.718638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a095853c-0dd5-45bb-ab46-174ed78525e2 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:38.723195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ce493-ba40-41dc-bc56-1aedf6e51649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:38.723819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ce493-ba40-41dc-bc56-1aedf6e51649 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.724609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ce493-ba40-41dc-bc56-1aedf6e51649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.724891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ce493-ba40-41dc-bc56-1aedf6e51649 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.725569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0ce493-ba40-41dc-bc56-1aedf6e51649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:38.727162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0ce493-ba40-41dc-bc56-1aedf6e51649 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:38.737355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1159/2304] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:38.741897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1b0cb8-812a-44bc-962a-0c59621dea33 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:38.751580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1b0cb8-812a-44bc-962a-0c59621dea33 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:38.756385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcfd1e4-9c0d-40c3-a4c0-3bea72060454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:38.756968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcfd1e4-9c0d-40c3-a4c0-3bea72060454 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.757242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcfd1e4-9c0d-40c3-a4c0-3bea72060454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.757511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcfd1e4-9c0d-40c3-a4c0-3bea72060454 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.758149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcfd1e4-9c0d-40c3-a4c0-3bea72060454 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:38.759727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcfd1e4-9c0d-40c3-a4c0-3bea72060454 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:38.770026 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1146/2305] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:12:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:12:38.774185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ce493-ba40-41dc-bc56-1aedf6e51649 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:38.784096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ce493-ba40-41dc-bc56-1aedf6e51649 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:38.789271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:38.789916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.790199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.790478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] PATH_INFO: `/v3/projects/6c7cce87cd0c4e6785bd0db3abe77f27/users/649ca2e37c934e4cb50e59bb4765b6b7/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.805453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c7cce87cd0c4e6785bd0db3abe77f27, user_id=649ca2e37c934e4cb50e59bb4765b6b7, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:38.808236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:38.821983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1160/2306] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:12:38 2022] PUT /identity/v3/projects/6c7cce87cd0c4e6785bd0db3abe77f27/users/649ca2e37c934e4cb50e59bb4765b6b7/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:12:38.827189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62f5713-2a0c-41b3-a49d-6c0aa8eb9913 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.827473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62f5713-2a0c-41b3-a49d-6c0aa8eb9913 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.827778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62f5713-2a0c-41b3-a49d-6c0aa8eb9913 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.845993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d62f5713-2a0c-41b3-a49d-6c0aa8eb9913 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:38.847186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d62f5713-2a0c-41b3-a49d-6c0aa8eb9913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:38.905009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1147/2307] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:38.915152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:38.916941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:38.972802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e12f90-e94b-4a20-96aa-8e198d243ed1 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:38.977164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:38.977765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:38.978045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:38.978316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:38.987262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:38.988715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:38.999764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1161/2308] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:39.753720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23012b64-0bca-4698-87b9-9ba2848b605c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:39.754148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23012b64-0bca-4698-87b9-9ba2848b605c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:39.754489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23012b64-0bca-4698-87b9-9ba2848b605c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:39.795605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-23012b64-0bca-4698-87b9-9ba2848b605c None None] MFA Rules not processed for user `649ca2e37c934e4cb50e59bb4765b6b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:39.796857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-23012b64-0bca-4698-87b9-9ba2848b605c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:39.817817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:39.865468 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1148/2309] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:39 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:39.876938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:12:39.878342 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:39.932369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:39.951590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ef6bf0-12ad-4e03-ad70-57d2cdf56f8f service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:12:39.955190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200374b4-9553-4712-ba1d-fbd15a719570 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:12:39.955848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200374b4-9553-4712-ba1d-fbd15a719570 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:39.956152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200374b4-9553-4712-ba1d-fbd15a719570 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:39.956430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200374b4-9553-4712-ba1d-fbd15a719570 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:39.966180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200374b4-9553-4712-ba1d-fbd15a719570 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:12:39.967640 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200374b4-9553-4712-ba1d-fbd15a719570 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:12:39.977793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1162/2310] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:12:39 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:12:41.355100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ecd18a-5820-4b59-8228-d435d5b82629 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:41.355620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ecd18a-5820-4b59-8228-d435d5b82629 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:41.355620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ecd18a-5820-4b59-8228-d435d5b82629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:41.374439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-09ecd18a-5820-4b59-8228-d435d5b82629 None None] MFA Rules not processed for user `649ca2e37c934e4cb50e59bb4765b6b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:41.375610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-09ecd18a-5820-4b59-8228-d435d5b82629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:41.382534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1149/2311] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:41 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:44.659515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ee05-5c23-4e12-989e-31cc609e1aa1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:12:44.660742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ee05-5c23-4e12-989e-31cc609e1aa1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:12:44.661455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ee05-5c23-4e12-989e-31cc609e1aa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:12:44.703111 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6304ee05-5c23-4e12-989e-31cc609e1aa1 None None] MFA Rules not processed for user `530e6587b39b4f4c98d7414b81453628`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:12:44.708971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6304ee05-5c23-4e12-989e-31cc609e1aa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:12:44.786375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1163/2312] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:12:44 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:12:51.369769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=20 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:12:54.774671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=33 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:21.660577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ecd18a-5820-4b59-8228-d435d5b82629 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:21.662211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-09ecd18a-5820-4b59-8228-d435d5b82629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:21.676906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ecd18a-5820-4b59-8228-d435d5b82629 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:21.681786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:21.682558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:21.682974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:21.683370 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] PATH_INFO: `/v3/users/434a2a9c6c6a473080a9bc84dfcc1e1e` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:21.684658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=434a2a9c6c6a473080a9bc84dfcc1e1e)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:21.686947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:21.749144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '434a2a9c6c6a473080a9bc84dfcc1e1e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:21.757167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '434a2a9c6c6a473080a9bc84dfcc1e1e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:21.757816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '434a2a9c6c6a473080a9bc84dfcc1e1e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:21.758529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '434a2a9c6c6a473080a9bc84dfcc1e1e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:21.765908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '434a2a9c6c6a473080a9bc84dfcc1e1e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:21.789229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1150/2313] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:13:21 2022] DELETE /identity/v3/users/434a2a9c6c6a473080a9bc84dfcc1e1e => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:13:21.794285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7051b26b-cfe0-4b2d-a82d-9c61aa95c234 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:21.794856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7051b26b-cfe0-4b2d-a82d-9c61aa95c234 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:21.794856 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7051b26b-cfe0-4b2d-a82d-9c61aa95c234 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:21.813617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7051b26b-cfe0-4b2d-a82d-9c61aa95c234 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:21.814802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7051b26b-cfe0-4b2d-a82d-9c61aa95c234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:21.873295 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1164/2314] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:21.882839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:21.884292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:21.940745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e3ceaf-54ae-4cb8-8b18-f194fc2601b0 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:21.944359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04bc05f-8144-4417-89bc-b766284d150c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:21.944958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04bc05f-8144-4417-89bc-b766284d150c service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:21.945237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04bc05f-8144-4417-89bc-b766284d150c service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:21.945510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04bc05f-8144-4417-89bc-b766284d150c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:21.954329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04bc05f-8144-4417-89bc-b766284d150c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:21.955777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04bc05f-8144-4417-89bc-b766284d150c service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:21.965661 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1151/2315] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:13:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:13:22.235297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7051b26b-cfe0-4b2d-a82d-9c61aa95c234 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:22.236909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7051b26b-cfe0-4b2d-a82d-9c61aa95c234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:22.252145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7051b26b-cfe0-4b2d-a82d-9c61aa95c234 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:22.257042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:22.257678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.257957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.258229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] PATH_INFO: `/v3/projects/19ce5c35248e4e58833cc3b7cfa703d2` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.259360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19ce5c35248e4e58833cc3b7cfa703d2)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:22.261620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:22.319073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19ce5c35248e4e58833cc3b7cfa703d2', 'request_id': 'req-d3b2a61c-fbb0-4bb1-8e22-598983554440'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:22.326668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19ce5c35248e4e58833cc3b7cfa703d2', 'request_id': 'req-d3b2a61c-fbb0-4bb1-8e22-598983554440'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:22.332341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19ce5c35248e4e58833cc3b7cfa703d2', 'request_id': 'req-d3b2a61c-fbb0-4bb1-8e22-598983554440'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:22.343844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] The token cache is being invalidate because project 19ce5c35248e4e58833cc3b7cfa703d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:13:22.344136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:22.344771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-d3b2a61c-fbb0-4bb1-8e22-598983554440 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:13:22.346264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1165/2316] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:13:22 2022] DELETE /identity/v3/projects/19ce5c35248e4e58833cc3b7cfa703d2 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:13:22.351209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2dffe1-9e1e-42bb-a62c-dc1fbd07139f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.351496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2dffe1-9e1e-42bb-a62c-dc1fbd07139f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.351800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2dffe1-9e1e-42bb-a62c-dc1fbd07139f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.371832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-0c2dffe1-9e1e-42bb-a62c-dc1fbd07139f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:22.373058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0c2dffe1-9e1e-42bb-a62c-dc1fbd07139f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:22.434327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1152/2317] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:22.443411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378227be-4641-412e-a668-2be97a699896 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.443712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378227be-4641-412e-a668-2be97a699896 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.444017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378227be-4641-412e-a668-2be97a699896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.463850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-378227be-4641-412e-a668-2be97a699896 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:22.465068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-378227be-4641-412e-a668-2be97a699896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:22.472927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1166/2318] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:22.476928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2dffe1-9e1e-42bb-a62c-dc1fbd07139f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:22.487099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2dffe1-9e1e-42bb-a62c-dc1fbd07139f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:22.491586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:22.492210 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.492492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.492766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.493416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:13:22.493718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:22.495376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:22.506585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1153/2319] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:13:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:13:22.511020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378227be-4641-412e-a668-2be97a699896 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:22.520272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378227be-4641-412e-a668-2be97a699896 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:22.524832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa2be9e-5567-4ace-85f0-458a32dae564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:22.525974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa2be9e-5567-4ace-85f0-458a32dae564 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.526443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa2be9e-5567-4ace-85f0-458a32dae564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.526883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa2be9e-5567-4ace-85f0-458a32dae564 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.527511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa2be9e-5567-4ace-85f0-458a32dae564 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:22.529448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa2be9e-5567-4ace-85f0-458a32dae564 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:22.556665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1167/2320] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:13:22 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:13:22.561265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:22.566398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:22.573049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1dd907-bd83-4f0e-9003-805fd0d7d01f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:22.630967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bca8951-0606-4564-b600-22d9bebab21b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:22.631576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bca8951-0606-4564-b600-22d9bebab21b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.631883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bca8951-0606-4564-b600-22d9bebab21b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.632151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bca8951-0606-4564-b600-22d9bebab21b admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.632775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bca8951-0606-4564-b600-22d9bebab21b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:22.634704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bca8951-0606-4564-b600-22d9bebab21b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:22.658008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1154/2321] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:13:22 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:13:22.662308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa2be9e-5567-4ace-85f0-458a32dae564 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:22.673249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa2be9e-5567-4ace-85f0-458a32dae564 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:22.678053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708f0d67-415b-47b2-86b6-e82c567ad4c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:22.678664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708f0d67-415b-47b2-86b6-e82c567ad4c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.678944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708f0d67-415b-47b2-86b6-e82c567ad4c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.679219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708f0d67-415b-47b2-86b6-e82c567ad4c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.680018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708f0d67-415b-47b2-86b6-e82c567ad4c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:22.682179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708f0d67-415b-47b2-86b6-e82c567ad4c8 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:22.693000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1168/2322] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:13:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:13:22.697020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bca8951-0606-4564-b600-22d9bebab21b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:22.707786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bca8951-0606-4564-b600-22d9bebab21b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:22.712456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b898000-b0b1-4dfc-ba6c-4539aecb4af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:22.713093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b898000-b0b1-4dfc-ba6c-4539aecb4af5 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.713421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b898000-b0b1-4dfc-ba6c-4539aecb4af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.713815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b898000-b0b1-4dfc-ba6c-4539aecb4af5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.714472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b898000-b0b1-4dfc-ba6c-4539aecb4af5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:22.716137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b898000-b0b1-4dfc-ba6c-4539aecb4af5 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:22.726173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1155/2323] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:13:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:13:22.730299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708f0d67-415b-47b2-86b6-e82c567ad4c8 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:22.740330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708f0d67-415b-47b2-86b6-e82c567ad4c8 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:22.746330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:22.746996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.747275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.747556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] PATH_INFO: `/v3/projects/2d56d6c8fce14f2f9fe424a5d0ae2046/users/2bf8e22b8bb2459b83d1c065c0da7145/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.765862 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d56d6c8fce14f2f9fe424a5d0ae2046, user_id=2bf8e22b8bb2459b83d1c065c0da7145, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:22.768568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:22.785586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1169/2324] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:13:22 2022] PUT /identity/v3/projects/2d56d6c8fce14f2f9fe424a5d0ae2046/users/2bf8e22b8bb2459b83d1c065c0da7145/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:13:22.790425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce06b056-08e8-4c38-b44c-2e7a32f3c86a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:22.790811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce06b056-08e8-4c38-b44c-2e7a32f3c86a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:22.791624 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce06b056-08e8-4c38-b44c-2e7a32f3c86a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:22.819441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-ce06b056-08e8-4c38-b44c-2e7a32f3c86a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:22.820929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ce06b056-08e8-4c38-b44c-2e7a32f3c86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:22.905546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1156/2325] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:13:22.918089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:22.919973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:22.949525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:23.009349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b3ecd7-ab8c-475a-a19d-a77d25bb764d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:23.019664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c3c813-ac71-4666-85be-550aa43d40dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:23.020502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c3c813-ac71-4666-85be-550aa43d40dd service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:23.020783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c3c813-ac71-4666-85be-550aa43d40dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:23.021055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c3c813-ac71-4666-85be-550aa43d40dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:23.032170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c3c813-ac71-4666-85be-550aa43d40dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:23.033627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c3c813-ac71-4666-85be-550aa43d40dd service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:23.044410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1170/2326] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:13:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:13:24.446342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce06b056-08e8-4c38-b44c-2e7a32f3c86a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:24.450012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-ce06b056-08e8-4c38-b44c-2e7a32f3c86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:24.529079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce06b056-08e8-4c38-b44c-2e7a32f3c86a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:24.534505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74478b2f-83f9-4edc-b587-a952c0ca3487 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:24.535395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74478b2f-83f9-4edc-b587-a952c0ca3487 service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:24.535890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74478b2f-83f9-4edc-b587-a952c0ca3487 service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:24.536425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74478b2f-83f9-4edc-b587-a952c0ca3487 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:24.538410 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-74478b2f-83f9-4edc-b587-a952c0ca3487 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:24.556123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74478b2f-83f9-4edc-b587-a952c0ca3487 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:24.558116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74478b2f-83f9-4edc-b587-a952c0ca3487 service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:24.572018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1157/2327] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:13:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:13:27.581436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601a0b25-94c6-4b40-b3b1-f31e8228cf0c None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:27.582002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601a0b25-94c6-4b40-b3b1-f31e8228cf0c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:27.582112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601a0b25-94c6-4b40-b3b1-f31e8228cf0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:27.607169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-601a0b25-94c6-4b40-b3b1-f31e8228cf0c None None] MFA Rules not processed for user `649ca2e37c934e4cb50e59bb4765b6b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:27.608503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-601a0b25-94c6-4b40-b3b1-f31e8228cf0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:27.685086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1171/2328] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:27 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:28.590849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bef8ab1-73d7-411b-9c53-029419211842 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:28.591318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bef8ab1-73d7-411b-9c53-029419211842 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:28.591477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bef8ab1-73d7-411b-9c53-029419211842 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:28.634196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:28.655459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-5bef8ab1-73d7-411b-9c53-029419211842 None None] MFA Rules not processed for user `2bf8e22b8bb2459b83d1c065c0da7145`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:28.656737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5bef8ab1-73d7-411b-9c53-029419211842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:28.748365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1158/2329] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:28.752775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601a0b25-94c6-4b40-b3b1-f31e8228cf0c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:28.771217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601a0b25-94c6-4b40-b3b1-f31e8228cf0c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:28.780907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6cbf5d-27d9-40ce-9271-5a2a7f14b54d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:28.790202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6cbf5d-27d9-40ce-9271-5a2a7f14b54d admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:28.790549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6cbf5d-27d9-40ce-9271-5a2a7f14b54d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:28.791530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6cbf5d-27d9-40ce-9271-5a2a7f14b54d admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:28.792620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6cbf5d-27d9-40ce-9271-5a2a7f14b54d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:28.804094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6cbf5d-27d9-40ce-9271-5a2a7f14b54d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:28.869580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1172/2330] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:13:28 2022] POST /identity/v3/projects => generated 383 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:13:28.870556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bef8ab1-73d7-411b-9c53-029419211842 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:28.887889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bef8ab1-73d7-411b-9c53-029419211842 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:28.967713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe99a8-df33-4e9a-b926-169dca7fed4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:28.968368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe99a8-df33-4e9a-b926-169dca7fed4b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:28.968758 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe99a8-df33-4e9a-b926-169dca7fed4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:28.968928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe99a8-df33-4e9a-b926-169dca7fed4b admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:28.969593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfe99a8-df33-4e9a-b926-169dca7fed4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:28.971529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfe99a8-df33-4e9a-b926-169dca7fed4b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:29.000669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1159/2331] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:13:28 2022] POST /identity/v3/users => generated 298 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:13:29.004324 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6cbf5d-27d9-40ce-9271-5a2a7f14b54d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:29.018361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6cbf5d-27d9-40ce-9271-5a2a7f14b54d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:29.023767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454f559-c00a-47fe-b41a-eac9da50cb33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:29.024393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454f559-c00a-47fe-b41a-eac9da50cb33 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:29.024855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454f559-c00a-47fe-b41a-eac9da50cb33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:29.025129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454f559-c00a-47fe-b41a-eac9da50cb33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:29.026004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2454f559-c00a-47fe-b41a-eac9da50cb33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:29.027636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2454f559-c00a-47fe-b41a-eac9da50cb33 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:29.042999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1173/2332] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:13:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:13:29.044858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe99a8-df33-4e9a-b926-169dca7fed4b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:29.056581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe99a8-df33-4e9a-b926-169dca7fed4b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:29.061837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dec17bc-c679-4468-9c8a-719603f693d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:29.062578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dec17bc-c679-4468-9c8a-719603f693d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:29.062916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dec17bc-c679-4468-9c8a-719603f693d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:29.063156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dec17bc-c679-4468-9c8a-719603f693d1 admin admin] PATH_INFO: `/v3/projects/7c1acf0de9e5442ca8a9a310de0238ed/users/88b6200f2c7f4236bdbcfe63a6fcff4c/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:29.080967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dec17bc-c679-4468-9c8a-719603f693d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c1acf0de9e5442ca8a9a310de0238ed, user_id=88b6200f2c7f4236bdbcfe63a6fcff4c, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:29.084002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dec17bc-c679-4468-9c8a-719603f693d1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:29.100369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1160/2333] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:13:29 2022] PUT /identity/v3/projects/7c1acf0de9e5442ca8a9a310de0238ed/users/88b6200f2c7f4236bdbcfe63a6fcff4c/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:13:29.104995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454f559-c00a-47fe-b41a-eac9da50cb33 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:29.115439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454f559-c00a-47fe-b41a-eac9da50cb33 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:29.175389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484197dc-f9cb-4cea-9823-05c3a9a24b83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:29.176052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484197dc-f9cb-4cea-9823-05c3a9a24b83 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:29.176338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484197dc-f9cb-4cea-9823-05c3a9a24b83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:29.176615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484197dc-f9cb-4cea-9823-05c3a9a24b83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:29.177272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484197dc-f9cb-4cea-9823-05c3a9a24b83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:29.178889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484197dc-f9cb-4cea-9823-05c3a9a24b83 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:29.192714 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1174/2334] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:13:29 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:13:29.193938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dec17bc-c679-4468-9c8a-719603f693d1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:29.204692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dec17bc-c679-4468-9c8a-719603f693d1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:29.211455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:29.216288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:29.216288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:29.216288 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] PATH_INFO: `/v3/projects/7c1acf0de9e5442ca8a9a310de0238ed/users/88b6200f2c7f4236bdbcfe63a6fcff4c/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:29.218792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c1acf0de9e5442ca8a9a310de0238ed, user_id=88b6200f2c7f4236bdbcfe63a6fcff4c, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:29.222312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:29.234802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1161/2335] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:13:29 2022] PUT /identity/v3/projects/7c1acf0de9e5442ca8a9a310de0238ed/users/88b6200f2c7f4236bdbcfe63a6fcff4c/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:13:35.645503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b7c69-7c72-48ad-a014-441f6c1050c8 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:35.646022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b7c69-7c72-48ad-a014-441f6c1050c8 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:35.646117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b7c69-7c72-48ad-a014-441f6c1050c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:35.674223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a71b7c69-7c72-48ad-a014-441f6c1050c8 None None] MFA Rules not processed for user `88b6200f2c7f4236bdbcfe63a6fcff4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:35.675433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a71b7c69-7c72-48ad-a014-441f6c1050c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:35.694687 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:35.745047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1175/2336] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:35 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:35.757105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:35.758818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:35.816994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1051a-3fbd-49b7-928b-02d82442f2c4 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:35.820556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '54db2e6805fd40f79c03eb3b2319f2b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:35.821157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:35.821436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:35.821706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:35.827334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:35.879567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:35.881016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:35.890484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1162/2337] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:13:35 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:13:36.033642 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a43608-fbfe-489d-abe7-157f0c494186 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:36.033936 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a43608-fbfe-489d-abe7-157f0c494186 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:36.034217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a43608-fbfe-489d-abe7-157f0c494186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:36.053569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-26a43608-fbfe-489d-abe7-157f0c494186 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:36.054832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-26a43608-fbfe-489d-abe7-157f0c494186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:36.113136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1176/2338] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:13:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:36.159806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:36.161271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:36.235744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998e2ff5-d253-46a0-8e24-7a267a64221d service glance] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:36.243385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0026063c-96ef-4e51-8b36-bb010389ea48 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:36.244849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0026063c-96ef-4e51-8b36-bb010389ea48 service swift] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:36.245186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0026063c-96ef-4e51-8b36-bb010389ea48 service swift] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:36.245456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0026063c-96ef-4e51-8b36-bb010389ea48 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:36.257110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0026063c-96ef-4e51-8b36-bb010389ea48 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:36.258834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0026063c-96ef-4e51-8b36-bb010389ea48 service swift] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:36.268990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1163/2339] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:13:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:13:38.545552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efde71b-dba7-4d6b-a9bb-366fc968e82a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:38.546177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efde71b-dba7-4d6b-a9bb-366fc968e82a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:38.546659 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efde71b-dba7-4d6b-a9bb-366fc968e82a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:38.597614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5efde71b-dba7-4d6b-a9bb-366fc968e82a None None] MFA Rules not processed for user `0ad0877d583d4550b9ab14ee7f4394f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:38.601071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5efde71b-dba7-4d6b-a9bb-366fc968e82a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:38.684239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1177/2340] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:38 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:46.265029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=56 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:46.667249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeeafb8-6d07-429d-911f-b415f2caaa50 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:46.667564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeeafb8-6d07-429d-911f-b415f2caaa50 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:46.667881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeeafb8-6d07-429d-911f-b415f2caaa50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:46.688216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-adeeafb8-6d07-429d-911f-b415f2caaa50 None None] MFA Rules not processed for user `2bf8e22b8bb2459b83d1c065c0da7145`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:46.689426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-adeeafb8-6d07-429d-911f-b415f2caaa50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:46.696225 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1164/2341] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:13:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:48.672939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=70 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:56.681173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:13:57.480842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b69e24-7b9b-4e47-8f0c-64fd731c50e3 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:57.481372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b69e24-7b9b-4e47-8f0c-64fd731c50e3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:57.481571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b69e24-7b9b-4e47-8f0c-64fd731c50e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:57.503136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e5b69e24-7b9b-4e47-8f0c-64fd731c50e3 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:13:57.504403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e5b69e24-7b9b-4e47-8f0c-64fd731c50e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:13:57.512497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1178/2342] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:13:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:13:57.524617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeeafb8-6d07-429d-911f-b415f2caaa50 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:13:57.538471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeeafb8-6d07-429d-911f-b415f2caaa50 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:13:57.544952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f591e9a-caaf-4f0a-96d5-0ecfe2dfad99 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:13:57.545712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f591e9a-caaf-4f0a-96d5-0ecfe2dfad99 service swift] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:13:57.546061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f591e9a-caaf-4f0a-96d5-0ecfe2dfad99 service swift] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:13:57.546406 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f591e9a-caaf-4f0a-96d5-0ecfe2dfad99 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:13:57.557584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f591e9a-caaf-4f0a-96d5-0ecfe2dfad99 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:13:57.560378 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f591e9a-caaf-4f0a-96d5-0ecfe2dfad99 service swift] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:13:57.572408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1165/2343] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:13:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 15 14:14:07.496649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:07.568769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:24.558062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427ea647-e461-4be6-b9b4-69f2d2903319 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:24.558667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427ea647-e461-4be6-b9b4-69f2d2903319 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:24.558667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427ea647-e461-4be6-b9b4-69f2d2903319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:24.579386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-427ea647-e461-4be6-b9b4-69f2d2903319 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:24.580565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-427ea647-e461-4be6-b9b4-69f2d2903319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:24.588214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1179/2344] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:14:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:24.600539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f591e9a-caaf-4f0a-96d5-0ecfe2dfad99 service swift] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:24.611888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f591e9a-caaf-4f0a-96d5-0ecfe2dfad99 service swift] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:24.617788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050bc69a-3a2c-4108-af5d-50b532e5c1f6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:24.618542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050bc69a-3a2c-4108-af5d-50b532e5c1f6 service swift] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:24.618909 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050bc69a-3a2c-4108-af5d-50b532e5c1f6 service swift] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:24.619239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050bc69a-3a2c-4108-af5d-50b532e5c1f6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:24.628112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050bc69a-3a2c-4108-af5d-50b532e5c1f6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:24.629677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050bc69a-3a2c-4108-af5d-50b532e5c1f6 service swift] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:24.638067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1166/2345] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:14:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:14:34.573355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:34.634276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:37.786791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427ea647-e461-4be6-b9b4-69f2d2903319 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:37.789413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-427ea647-e461-4be6-b9b4-69f2d2903319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:37.854679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427ea647-e461-4be6-b9b4-69f2d2903319 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:37.858758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1690c677-edc1-40b1-b19b-725165cf11fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:37.859445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690c677-edc1-40b1-b19b-725165cf11fe service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:37.859829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690c677-edc1-40b1-b19b-725165cf11fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:37.860101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1690c677-edc1-40b1-b19b-725165cf11fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:37.864273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1690c677-edc1-40b1-b19b-725165cf11fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:37.947006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1690c677-edc1-40b1-b19b-725165cf11fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:37.948489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1690c677-edc1-40b1-b19b-725165cf11fe service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:37.961254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1180/2346] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:14:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:14:38.378192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050bc69a-3a2c-4108-af5d-50b532e5c1f6 service swift] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:38.388369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050bc69a-3a2c-4108-af5d-50b532e5c1f6 service swift] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:38.392309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:38.393105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:38.393384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:38.393653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:38.397799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:38.412713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:38.414163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:38.426942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1167/2347] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:14:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:14:43.688227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aeb0aa-7864-49d8-b3fa-f1ffb0aa6ce4 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:43.688715 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aeb0aa-7864-49d8-b3fa-f1ffb0aa6ce4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:43.688820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aeb0aa-7864-49d8-b3fa-f1ffb0aa6ce4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:43.710491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-87aeb0aa-7864-49d8-b3fa-f1ffb0aa6ce4 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:43.711708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-87aeb0aa-7864-49d8-b3fa-f1ffb0aa6ce4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:43.719348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1181/2348] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 15 14:14:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:43.732022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:43.743247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f48875-679a-4d33-be24-749bdcecac27 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:43.748120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607f43c5-9f26-491b-8f38-b5ef0ca9b550 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:43.748709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607f43c5-9f26-491b-8f38-b5ef0ca9b550 service swift] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:43.748990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607f43c5-9f26-491b-8f38-b5ef0ca9b550 service swift] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:43.750068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607f43c5-9f26-491b-8f38-b5ef0ca9b550 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:43.757899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607f43c5-9f26-491b-8f38-b5ef0ca9b550 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:43.759437 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607f43c5-9f26-491b-8f38-b5ef0ca9b550 service swift] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:43.767643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1168/2349] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:14:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:14:45.906570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a2db2-89a9-41a5-95bc-c42da235484f None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:45.906570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a2db2-89a9-41a5-95bc-c42da235484f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:45.906570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3a2db2-89a9-41a5-95bc-c42da235484f None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:45.906570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1182/2350] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:14:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:14:45.940301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c06de4-7d44-41bd-b721-695ab6fb562d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:45.940787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c06de4-7d44-41bd-b721-695ab6fb562d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:45.940877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c06de4-7d44-41bd-b721-695ab6fb562d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:45.988749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-33c06de4-7d44-41bd-b721-695ab6fb562d None None] MFA Rules not processed for user `62299b21a55c46d78a62098ee9e3c497`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:45.990292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-33c06de4-7d44-41bd-b721-695ab6fb562d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:46.007059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3a2db2-89a9-41a5-95bc-c42da235484f None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:46.013033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ed3a2db2-89a9-41a5-95bc-c42da235484f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:46.079208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1169/2351] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:14:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:14:46.100309 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c06de4-7d44-41bd-b721-695ab6fb562d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:46.117493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3a2db2-89a9-41a5-95bc-c42da235484f None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:46.119559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c06de4-7d44-41bd-b721-695ab6fb562d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:46.124722 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c536ec-c1d9-42bb-bb61-a8efd3245212 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:46.125621 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c536ec-c1d9-42bb-bb61-a8efd3245212 service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:46.126868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c536ec-c1d9-42bb-bb61-a8efd3245212 service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:46.127399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c536ec-c1d9-42bb-bb61-a8efd3245212 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:46.131746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:46.132636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:46.132923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:46.133322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] PATH_INFO: `/v3/users/649ca2e37c934e4cb50e59bb4765b6b7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:46.135123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=649ca2e37c934e4cb50e59bb4765b6b7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:46.137296 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:46.142176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c536ec-c1d9-42bb-bb61-a8efd3245212 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:46.145087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c536ec-c1d9-42bb-bb61-a8efd3245212 service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:46.171118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1170/2352] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:14:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:14:46.210155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:46.223178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '649ca2e37c934e4cb50e59bb4765b6b7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.233142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '649ca2e37c934e4cb50e59bb4765b6b7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.233819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '649ca2e37c934e4cb50e59bb4765b6b7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.234808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '649ca2e37c934e4cb50e59bb4765b6b7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.243436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3d9354-b26f-4bca-88dc-8d8d2ffdcc8d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:46.244256 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3d9354-b26f-4bca-88dc-8d8d2ffdcc8d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:46.244733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3d9354-b26f-4bca-88dc-8d8d2ffdcc8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:46.245130 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '649ca2e37c934e4cb50e59bb4765b6b7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.264382 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1183/2353] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:14:46 2022] DELETE /identity/v3/users/649ca2e37c934e4cb50e59bb4765b6b7 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:14:46.267069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-fe3d9354-b26f-4bca-88dc-8d8d2ffdcc8d None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:46.268497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fe3d9354-b26f-4bca-88dc-8d8d2ffdcc8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:46.350900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1171/2354] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:14:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:14:46.364976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:46.366747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:46.473371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf78cd6-b214-482d-9c11-c1249a84ec33 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:46.490531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dcb864-5b78-4715-97a3-b054c3ae5ac6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:46.491519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dcb864-5b78-4715-97a3-b054c3ae5ac6 service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:46.492132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dcb864-5b78-4715-97a3-b054c3ae5ac6 service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:46.492378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dcb864-5b78-4715-97a3-b054c3ae5ac6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:46.510949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dcb864-5b78-4715-97a3-b054c3ae5ac6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:46.512896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dcb864-5b78-4715-97a3-b054c3ae5ac6 service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:46.528128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1184/2355] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:14:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:14:46.584667 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3d9354-b26f-4bca-88dc-8d8d2ffdcc8d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:46.586160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fe3d9354-b26f-4bca-88dc-8d8d2ffdcc8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:46.700190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3d9354-b26f-4bca-88dc-8d8d2ffdcc8d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:46.708461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:46.710130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:46.710414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:46.710900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] PATH_INFO: `/v3/projects/6c7cce87cd0c4e6785bd0db3abe77f27` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:46.712655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c7cce87cd0c4e6785bd0db3abe77f27)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:46.715778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:46.722595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:46.789622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c7cce87cd0c4e6785bd0db3abe77f27', 'request_id': 'req-5976e5db-73af-473d-933a-304813b1408e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.799020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c7cce87cd0c4e6785bd0db3abe77f27', 'request_id': 'req-5976e5db-73af-473d-933a-304813b1408e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.807859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c7cce87cd0c4e6785bd0db3abe77f27', 'request_id': 'req-5976e5db-73af-473d-933a-304813b1408e'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.824848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] The token cache is being invalidate because project 6c7cce87cd0c4e6785bd0db3abe77f27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:14:46.825221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.826011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5976e5db-73af-473d-933a-304813b1408e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:46.828131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1172/2356] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:14:46 2022] DELETE /identity/v3/projects/6c7cce87cd0c4e6785bd0db3abe77f27 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:14:46.834343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56ef48-7b10-430f-9ba3-42f2f17cf850 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:46.834536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56ef48-7b10-430f-9ba3-42f2f17cf850 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:46.834970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56ef48-7b10-430f-9ba3-42f2f17cf850 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:46.862069 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7f56ef48-7b10-430f-9ba3-42f2f17cf850 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:46.863396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7f56ef48-7b10-430f-9ba3-42f2f17cf850 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:46.974245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1185/2357] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:14:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:46.986591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecd35ea-f67d-4ca3-840b-3883daa0cd01 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:46.987120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecd35ea-f67d-4ca3-840b-3883daa0cd01 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:46.987249 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecd35ea-f67d-4ca3-840b-3883daa0cd01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.011542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2ecd35ea-f67d-4ca3-840b-3883daa0cd01 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:47.013009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2ecd35ea-f67d-4ca3-840b-3883daa0cd01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:47.022073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1173/2358] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:14:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:47.026053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f56ef48-7b10-430f-9ba3-42f2f17cf850 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:47.038391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f56ef48-7b10-430f-9ba3-42f2f17cf850 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:47.046807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:47.048462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:47.048462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:47.048462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.049197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:14:47.049632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:47.051380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:47.065566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1186/2359] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:14:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:14:47.069740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecd35ea-f67d-4ca3-840b-3883daa0cd01 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:47.080237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecd35ea-f67d-4ca3-840b-3883daa0cd01 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:47.085434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dd4219-b4a0-476b-a46b-9d66ce99f215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:47.085974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dd4219-b4a0-476b-a46b-9d66ce99f215 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:47.086262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dd4219-b4a0-476b-a46b-9d66ce99f215 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:47.086523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dd4219-b4a0-476b-a46b-9d66ce99f215 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.087172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dd4219-b4a0-476b-a46b-9d66ce99f215 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:47.089187 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dd4219-b4a0-476b-a46b-9d66ce99f215 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:47.124851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:47.126783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1174/2360] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:14:47 2022] POST /identity/v3/projects => generated 377 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:14:47.142083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9957eba-7af2-41d9-9988-b8e78ac0ab74 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:47.179413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:47.221675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365122d-8547-4a36-97b7-a9c8fe9abdad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:47.222204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365122d-8547-4a36-97b7-a9c8fe9abdad admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:47.222484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365122d-8547-4a36-97b7-a9c8fe9abdad admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:47.222757 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365122d-8547-4a36-97b7-a9c8fe9abdad admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.223494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0365122d-8547-4a36-97b7-a9c8fe9abdad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:47.225788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0365122d-8547-4a36-97b7-a9c8fe9abdad admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:47.253754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1187/2361] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:14:47 2022] POST /identity/v3/users => generated 289 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:14:47.257872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dd4219-b4a0-476b-a46b-9d66ce99f215 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:47.268641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dd4219-b4a0-476b-a46b-9d66ce99f215 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:47.273946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcd3f17-187a-4b7f-b83d-0a428127fd3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:47.274558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcd3f17-187a-4b7f-b83d-0a428127fd3e admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:47.274837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcd3f17-187a-4b7f-b83d-0a428127fd3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:47.275108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcd3f17-187a-4b7f-b83d-0a428127fd3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.275784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcd3f17-187a-4b7f-b83d-0a428127fd3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:47.277382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcd3f17-187a-4b7f-b83d-0a428127fd3e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:47.291881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1175/2362] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:14:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:14:47.293486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365122d-8547-4a36-97b7-a9c8fe9abdad admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:47.319452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365122d-8547-4a36-97b7-a9c8fe9abdad admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:47.324909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f70c8f4-48f1-4e74-b653-83c1adb97184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:47.325817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f70c8f4-48f1-4e74-b653-83c1adb97184 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:47.326103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f70c8f4-48f1-4e74-b653-83c1adb97184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:47.326379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f70c8f4-48f1-4e74-b653-83c1adb97184 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.327028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f70c8f4-48f1-4e74-b653-83c1adb97184 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:47.329327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f70c8f4-48f1-4e74-b653-83c1adb97184 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:47.345662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1188/2363] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:14:47 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:14:47.349906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcd3f17-187a-4b7f-b83d-0a428127fd3e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:47.367553 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcd3f17-187a-4b7f-b83d-0a428127fd3e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:47.374095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:47.374780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:47.375059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:47.375421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] PATH_INFO: `/v3/projects/45493bb87bce44b98183b36c4aa9751c/users/cc8e5336079d4b4b92283febb3618927/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.402112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=45493bb87bce44b98183b36c4aa9751c, user_id=cc8e5336079d4b4b92283febb3618927, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:47.408149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:47.434475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1176/2364] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:14:47 2022] PUT /identity/v3/projects/45493bb87bce44b98183b36c4aa9751c/users/cc8e5336079d4b4b92283febb3618927/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:14:47.521434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50b615-1f92-4463-9af0-492d60df2236 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:47.521928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50b615-1f92-4463-9af0-492d60df2236 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:47.522211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50b615-1f92-4463-9af0-492d60df2236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.581569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2e50b615-1f92-4463-9af0-492d60df2236 None None] MFA Rules not processed for user `cc8e5336079d4b4b92283febb3618927`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:47.594544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2e50b615-1f92-4463-9af0-492d60df2236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:47.742966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1189/2365] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:14:47 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:47.757958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:47.760451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:47.901575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58b6e02-9429-4187-9358-a48a6e1c70bc admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:47.910768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8636a6c0-9704-4da8-9745-d076813d25de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:47.911387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8636a6c0-9704-4da8-9745-d076813d25de service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:47.911664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8636a6c0-9704-4da8-9745-d076813d25de service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:47.911964 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8636a6c0-9704-4da8-9745-d076813d25de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:47.930884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8636a6c0-9704-4da8-9745-d076813d25de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:47.932558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8636a6c0-9704-4da8-9745-d076813d25de service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:47.943811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1177/2366] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:14:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:14:50.832973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e50b615-1f92-4463-9af0-492d60df2236 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:50.912833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e50b615-1f92-4463-9af0-492d60df2236 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:50.917712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:50.918450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:50.918805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:50.919151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] PATH_INFO: `/v3/users/cc8e5336079d4b4b92283febb3618927` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:50.920357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc8e5336079d4b4b92283febb3618927)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:50.923127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:50.975645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:50.984262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc8e5336079d4b4b92283febb3618927'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:50.996372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc8e5336079d4b4b92283febb3618927'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:50.997240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc8e5336079d4b4b92283febb3618927'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:50.998045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc8e5336079d4b4b92283febb3618927'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:51.007335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc8e5336079d4b4b92283febb3618927'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:51.036846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1190/2367] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:14:50 2022] DELETE /identity/v3/users/cc8e5336079d4b4b92283febb3618927 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:14:51.038631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47d2f81-4c84-4794-a8c4-7ac69c284368 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:51.040500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47d2f81-4c84-4794-a8c4-7ac69c284368 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:51.040500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47d2f81-4c84-4794-a8c4-7ac69c284368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:51.060582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f47d2f81-4c84-4794-a8c4-7ac69c284368 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:51.061709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f47d2f81-4c84-4794-a8c4-7ac69c284368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:51.091721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:51.134567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1178/2368] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:14:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:51.146353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:51.147992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:51.218129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733f3200-e429-41b1-bb30-786dae2fdef5 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:51.222767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7316445-338d-42a8-b614-de2854226247 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:51.223560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7316445-338d-42a8-b614-de2854226247 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:51.223815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7316445-338d-42a8-b614-de2854226247 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:51.224270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7316445-338d-42a8-b614-de2854226247 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:51.238062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7316445-338d-42a8-b614-de2854226247 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:51.239574 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7316445-338d-42a8-b614-de2854226247 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:51.250710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1191/2369] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:14:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:14:51.677774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47d2f81-4c84-4794-a8c4-7ac69c284368 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:51.679233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f47d2f81-4c84-4794-a8c4-7ac69c284368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:51.693347 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47d2f81-4c84-4794-a8c4-7ac69c284368 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:51.702932 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:51.704726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:51.705976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:51.706313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] PATH_INFO: `/v3/projects/45493bb87bce44b98183b36c4aa9751c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:51.708445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] RBAC: Authorizing `identity:delete_project(project_id=45493bb87bce44b98183b36c4aa9751c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:51.710865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:51.788042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45493bb87bce44b98183b36c4aa9751c', 'request_id': 'req-26ba8791-6ef2-4d65-8f53-b32409f4987b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:51.800745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45493bb87bce44b98183b36c4aa9751c', 'request_id': 'req-26ba8791-6ef2-4d65-8f53-b32409f4987b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:51.812041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45493bb87bce44b98183b36c4aa9751c', 'request_id': 'req-26ba8791-6ef2-4d65-8f53-b32409f4987b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:51.832258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] The token cache is being invalidate because project 45493bb87bce44b98183b36c4aa9751c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:14:51.832464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:51.833135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-26ba8791-6ef2-4d65-8f53-b32409f4987b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:14:51.834834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1179/2370] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:14:51 2022] DELETE /identity/v3/projects/45493bb87bce44b98183b36c4aa9751c => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:14:51.840842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39bd8ba-85db-4c99-989d-7d350a7365e5 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:51.842814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39bd8ba-85db-4c99-989d-7d350a7365e5 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:51.843317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39bd8ba-85db-4c99-989d-7d350a7365e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:51.870128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c39bd8ba-85db-4c99-989d-7d350a7365e5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:51.871891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c39bd8ba-85db-4c99-989d-7d350a7365e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:51.963928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1192/2371] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:14:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:51.972425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199e586f-e992-41cf-b0e9-05aa343268e4 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:51.972709 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199e586f-e992-41cf-b0e9-05aa343268e4 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:51.972991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199e586f-e992-41cf-b0e9-05aa343268e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:51.997085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-199e586f-e992-41cf-b0e9-05aa343268e4 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:51.998310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-199e586f-e992-41cf-b0e9-05aa343268e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:52.011351 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1180/2372] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:14:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:52.016877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39bd8ba-85db-4c99-989d-7d350a7365e5 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:52.027151 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39bd8ba-85db-4c99-989d-7d350a7365e5 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:52.032552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:52.033348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:52.034051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:52.034051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:52.034919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:14:52.035353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:52.039778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:52.058265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1193/2373] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:14:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:14:52.059363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199e586f-e992-41cf-b0e9-05aa343268e4 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:52.070147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199e586f-e992-41cf-b0e9-05aa343268e4 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:52.074939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cfd413-2adb-4399-9b31-47de6fe674eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:52.075563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cfd413-2adb-4399-9b31-47de6fe674eb admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:52.075865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cfd413-2adb-4399-9b31-47de6fe674eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:52.076142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cfd413-2adb-4399-9b31-47de6fe674eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:52.076789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cfd413-2adb-4399-9b31-47de6fe674eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:52.080335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cfd413-2adb-4399-9b31-47de6fe674eb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:52.115376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1181/2374] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:14:52 2022] POST /identity/v3/projects => generated 375 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:14:52.120074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:52.132868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad45a29-0cbe-4c87-b78a-c0304a16c440 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:52.183549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:52.199807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e730c84-c28e-406c-9c2b-5de81d7dc018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:52.200426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e730c84-c28e-406c-9c2b-5de81d7dc018 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:52.200762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e730c84-c28e-406c-9c2b-5de81d7dc018 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:52.201034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e730c84-c28e-406c-9c2b-5de81d7dc018 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:52.201674 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e730c84-c28e-406c-9c2b-5de81d7dc018 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:52.203646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e730c84-c28e-406c-9c2b-5de81d7dc018 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:52.230199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1194/2375] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:14:52 2022] POST /identity/v3/users => generated 288 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:14:52.231377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cfd413-2adb-4399-9b31-47de6fe674eb admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:52.245073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cfd413-2adb-4399-9b31-47de6fe674eb admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:52.251010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb13b2c-87df-4ee3-bd4e-4d45762a313c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:52.251703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb13b2c-87df-4ee3-bd4e-4d45762a313c admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:52.251988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb13b2c-87df-4ee3-bd4e-4d45762a313c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:52.252284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb13b2c-87df-4ee3-bd4e-4d45762a313c admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:52.252983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb13b2c-87df-4ee3-bd4e-4d45762a313c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:52.254977 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb13b2c-87df-4ee3-bd4e-4d45762a313c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:52.266805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1182/2376] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:14:52 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:14:52.270967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e730c84-c28e-406c-9c2b-5de81d7dc018 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:52.281144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e730c84-c28e-406c-9c2b-5de81d7dc018 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:52.286654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3a7d00-84b9-49f0-8150-1dc4486eaca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:52.287267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a7d00-84b9-49f0-8150-1dc4486eaca5 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:52.287548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a7d00-84b9-49f0-8150-1dc4486eaca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:52.287850 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a7d00-84b9-49f0-8150-1dc4486eaca5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:52.288497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3a7d00-84b9-49f0-8150-1dc4486eaca5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:52.290114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3a7d00-84b9-49f0-8150-1dc4486eaca5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:52.307435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1195/2377] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:14:52 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:14:52.311967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb13b2c-87df-4ee3-bd4e-4d45762a313c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:52.327511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb13b2c-87df-4ee3-bd4e-4d45762a313c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:52.333165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:52.333895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:52.334280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:52.334620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] PATH_INFO: `/v3/projects/8e8d59efe51f47c2ad06206d0039db7c/users/733f559b92164359a373f0d46554faf2/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:52.351251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e8d59efe51f47c2ad06206d0039db7c, user_id=733f559b92164359a373f0d46554faf2, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:52.354037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:52.369650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1183/2378] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:14:52 2022] PUT /identity/v3/projects/8e8d59efe51f47c2ad06206d0039db7c/users/733f559b92164359a373f0d46554faf2/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:14:52.382330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90cf1bb-1c7b-4075-afed-42fbf761b957 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:52.383118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90cf1bb-1c7b-4075-afed-42fbf761b957 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:52.386378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90cf1bb-1c7b-4075-afed-42fbf761b957 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:52.422533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e90cf1bb-1c7b-4075-afed-42fbf761b957 None None] MFA Rules not processed for user `733f559b92164359a373f0d46554faf2`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:14:52.423851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e90cf1bb-1c7b-4075-afed-42fbf761b957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:52.515598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1196/2379] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:14:52 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:14:52.527317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:52.529160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:52.600505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa147b-1c0e-4cc4-a45a-e7e9478ad6a4 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:52.605310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:52.606165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:52.606655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:52.607159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:52.613487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:14:52.621677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:52.623401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:52.637855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1184/2380] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:14:52 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:14:57.040585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90cf1bb-1c7b-4075-afed-42fbf761b957 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:14:57.042098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e90cf1bb-1c7b-4075-afed-42fbf761b957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:57.129737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90cf1bb-1c7b-4075-afed-42fbf761b957 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:14:57.133751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '54db2e6805fd40f79c03eb3b2319f2b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:14:57.134335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:14:57.134577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:14:57.134844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:14:57.136678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:14:57.201065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:14:57.202513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:14:57.213299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1197/2381] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:14:57 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:02.632141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:04.605320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:04.606844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:04.668820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30457860-55ca-47f8-bfac-a420e17c306d service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:04.673554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f238eab-ffd8-4c41-ab47-8ede6eae22b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:04.674575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f238eab-ffd8-4c41-ab47-8ede6eae22b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:04.674988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f238eab-ffd8-4c41-ab47-8ede6eae22b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:04.675265 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f238eab-ffd8-4c41-ab47-8ede6eae22b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:04.677173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9f238eab-ffd8-4c41-ab47-8ede6eae22b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:04.760440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f238eab-ffd8-4c41-ab47-8ede6eae22b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:04.761976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f238eab-ffd8-4c41-ab47-8ede6eae22b8 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:04.774492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1185/2382] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:04 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:05.400412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:05.413002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:05.467381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e43e07-7edb-478e-a321-cb6d5bda5b3f service glance] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:05.472931 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:05.473565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:05.473964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:05.474244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] PATH_INFO: `/v3/users/733f559b92164359a373f0d46554faf2` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:05.475368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=733f559b92164359a373f0d46554faf2)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:05.477348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:05.527355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '733f559b92164359a373f0d46554faf2'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:05.536381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '733f559b92164359a373f0d46554faf2'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:05.536999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '733f559b92164359a373f0d46554faf2'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:05.537736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '733f559b92164359a373f0d46554faf2'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:05.544839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '733f559b92164359a373f0d46554faf2'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:05.563822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1198/2383] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:15:05 2022] DELETE /identity/v3/users/733f559b92164359a373f0d46554faf2 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:15:05.568526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb31f90a-9dcf-425c-a1ac-7c2108bc6603 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:05.568961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb31f90a-9dcf-425c-a1ac-7c2108bc6603 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:05.569340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb31f90a-9dcf-425c-a1ac-7c2108bc6603 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:05.589535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-bb31f90a-9dcf-425c-a1ac-7c2108bc6603 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:05.590814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bb31f90a-9dcf-425c-a1ac-7c2108bc6603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:05.649509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1186/2384] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:15:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:05.660118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:05.661650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:05.718996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ad1363-55ef-4ae8-8880-687b06194c2b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:05.723057 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:05.723707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:05.724000 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:05.724279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:05.733616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:05.735049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:05.744801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1199/2385] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:06.052055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb31f90a-9dcf-425c-a1ac-7c2108bc6603 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:06.054640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bb31f90a-9dcf-425c-a1ac-7c2108bc6603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:06.094907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb31f90a-9dcf-425c-a1ac-7c2108bc6603 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:06.101160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:06.102230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:06.102567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:06.102845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] PATH_INFO: `/v3/projects/8e8d59efe51f47c2ad06206d0039db7c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:06.104429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e8d59efe51f47c2ad06206d0039db7c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:06.107825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:06.180272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:06.183717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e8d59efe51f47c2ad06206d0039db7c', 'request_id': 'req-5612e4aa-846e-4824-898b-350641c0a604'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:06.195420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e8d59efe51f47c2ad06206d0039db7c', 'request_id': 'req-5612e4aa-846e-4824-898b-350641c0a604'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:06.206679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e8d59efe51f47c2ad06206d0039db7c', 'request_id': 'req-5612e4aa-846e-4824-898b-350641c0a604'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:06.222723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] The token cache is being invalidate because project 8e8d59efe51f47c2ad06206d0039db7c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:15:06.223339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:06.224192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5612e4aa-846e-4824-898b-350641c0a604 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:06.226109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1187/2386] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:15:06 2022] DELETE /identity/v3/projects/8e8d59efe51f47c2ad06206d0039db7c => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:15:08.172078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:08.174944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:08.186543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1d585-9fad-462e-96e8-65a92bde71df service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:08.219742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:08.250108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ef78e4-768c-498e-9314-b583725b4359 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:08.250904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ef78e4-768c-498e-9314-b583725b4359 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.251248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ef78e4-768c-498e-9314-b583725b4359 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.252334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ef78e4-768c-498e-9314-b583725b4359 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.254267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b7ef78e4-768c-498e-9314-b583725b4359 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:08.264431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ef78e4-768c-498e-9314-b583725b4359 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:08.265972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ef78e4-768c-498e-9314-b583725b4359 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:08.275884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1200/2387] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:08.408776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd05ef-5603-4a81-beab-f83ac4dfcb3f None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.409655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd05ef-5603-4a81-beab-f83ac4dfcb3f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.409935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd05ef-5603-4a81-beab-f83ac4dfcb3f None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.411285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1188/2388] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:15:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:15:08.416959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23826f4-218f-48af-9058-7eadf6704c4a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.417327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23826f4-218f-48af-9058-7eadf6704c4a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.417698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23826f4-218f-48af-9058-7eadf6704c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.440983 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c23826f4-218f-48af-9058-7eadf6704c4a None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:08.442371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c23826f4-218f-48af-9058-7eadf6704c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:08.527482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1201/2389] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:15:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:15:08.543131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd05ef-5603-4a81-beab-f83ac4dfcb3f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:08.545328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-91dd05ef-5603-4a81-beab-f83ac4dfcb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:08.616008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd05ef-5603-4a81-beab-f83ac4dfcb3f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:08.619975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80af26a9-292c-4a87-9f89-1f984894f12f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '54db2e6805fd40f79c03eb3b2319f2b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:08.620603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af26a9-292c-4a87-9f89-1f984894f12f service glance] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.620888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af26a9-292c-4a87-9f89-1f984894f12f service glance] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.621207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af26a9-292c-4a87-9f89-1f984894f12f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.633613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80af26a9-292c-4a87-9f89-1f984894f12f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:08.635137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80af26a9-292c-4a87-9f89-1f984894f12f service glance] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:08.647492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1189/2390] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:08.741756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005b5716-a4f0-4f62-84a4-75506e44124b None None] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.742159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005b5716-a4f0-4f62-84a4-75506e44124b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.742440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005b5716-a4f0-4f62-84a4-75506e44124b None None] PATH_INFO: `/` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.743907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1202/2391] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:15:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:15:08.748695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f592fc2-750f-45ba-9bc4-60d092fba7ae None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.749008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f592fc2-750f-45ba-9bc4-60d092fba7ae None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.749282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f592fc2-750f-45ba-9bc4-60d092fba7ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.769941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-2f592fc2-750f-45ba-9bc4-60d092fba7ae None None] MFA Rules not processed for user `922012eacf244f9480c180c2eb9eaf28`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:08.771150 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2f592fc2-750f-45ba-9bc4-60d092fba7ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:08.777866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1190/2392] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:15:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:08.792633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005b5716-a4f0-4f62-84a4-75506e44124b None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:08.794369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-005b5716-a4f0-4f62-84a4-75506e44124b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:08.808785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005b5716-a4f0-4f62-84a4-75506e44124b None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:08.815617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d795893-da2b-485b-bd47-a80b3fdceb20 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:08.815617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d795893-da2b-485b-bd47-a80b3fdceb20 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.815617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d795893-da2b-485b-bd47-a80b3fdceb20 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.815617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d795893-da2b-485b-bd47-a80b3fdceb20 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.822836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d795893-da2b-485b-bd47-a80b3fdceb20 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:08.824440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d795893-da2b-485b-bd47-a80b3fdceb20 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:08.832668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1203/2393] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:08.919048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe3b7d-0303-42ff-8c4b-9cadde7956be None None] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.919352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe3b7d-0303-42ff-8c4b-9cadde7956be None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.919622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe3b7d-0303-42ff-8c4b-9cadde7956be None None] PATH_INFO: `/` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.921019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1191/2394] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 15 14:15:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 15 14:15:08.926960 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913359a-2882-42ed-8708-723c46f52173 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.927243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913359a-2882-42ed-8708-723c46f52173 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.927518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913359a-2882-42ed-8708-723c46f52173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.947518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a913359a-2882-42ed-8708-723c46f52173 None None] MFA Rules not processed for user `ce6744790db74f0aaffc6eb269bf6737`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:08.948733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a913359a-2882-42ed-8708-723c46f52173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:08.958217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1204/2395] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 15 14:15:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:08.968443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fe3b7d-0303-42ff-8c4b-9cadde7956be None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:08.979820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fe3b7d-0303-42ff-8c4b-9cadde7956be None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:08.983349 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:08.984038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:08.984318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:08.984588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:08.995233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:08.996611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:09.007034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1192/2396] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:09.070874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a913359a-2882-42ed-8708-723c46f52173 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:09.072316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a913359a-2882-42ed-8708-723c46f52173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:09.140416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a913359a-2882-42ed-8708-723c46f52173 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:09.146492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cecd1ba-ee3b-47c9-9984-cb616524f60b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:09.147110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecd1ba-ee3b-47c9-9984-cb616524f60b service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:09.147393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecd1ba-ee3b-47c9-9984-cb616524f60b service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:09.147670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecd1ba-ee3b-47c9-9984-cb616524f60b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:09.149498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5cecd1ba-ee3b-47c9-9984-cb616524f60b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:09.161538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cecd1ba-ee3b-47c9-9984-cb616524f60b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:09.163117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cecd1ba-ee3b-47c9-9984-cb616524f60b service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:09.172859 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1205/2397] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:12.974208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:12.975703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:12.987325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea2f2ab-e742-4671-bef5-56212475eeb0 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:12.991022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:12.991617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:12.991970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:12.992248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:12.993941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:13.004556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:13.005970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:13.014441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1193/2398] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:15.413707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31015fcd-8cb8-4628-8a93-f9a7496ccdd4 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:15.414205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31015fcd-8cb8-4628-8a93-f9a7496ccdd4 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:15.414301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31015fcd-8cb8-4628-8a93-f9a7496ccdd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:15.434546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-31015fcd-8cb8-4628-8a93-f9a7496ccdd4 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:15.435784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-31015fcd-8cb8-4628-8a93-f9a7496ccdd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:15.475620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:15.501056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1206/2399] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:15:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:15.516272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:15.516272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:15.586419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad15e0-117a-4ff8-88b1-b291523a81a5 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:15.591401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e359080a-ef21-48f3-aaba-795b376bff4a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:15.592041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e359080a-ef21-48f3-aaba-795b376bff4a service swift] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:15.592324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e359080a-ef21-48f3-aaba-795b376bff4a service swift] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:15.592597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e359080a-ef21-48f3-aaba-795b376bff4a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:15.601234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e359080a-ef21-48f3-aaba-795b376bff4a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:15.602807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e359080a-ef21-48f3-aaba-795b376bff4a service swift] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:15.610749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1194/2400] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:15:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 15 14:15:24.102811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31015fcd-8cb8-4628-8a93-f9a7496ccdd4 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:24.104393 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-31015fcd-8cb8-4628-8a93-f9a7496ccdd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:24.201137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31015fcd-8cb8-4628-8a93-f9a7496ccdd4 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:24.207042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:24.207827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:24.208234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:24.208522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] PATH_INFO: `/v3/users/0ad0877d583d4550b9ab14ee7f4394f0` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:24.209770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ad0877d583d4550b9ab14ee7f4394f0)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:24.211846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:24.271386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ad0877d583d4550b9ab14ee7f4394f0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.278582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ad0877d583d4550b9ab14ee7f4394f0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.279191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ad0877d583d4550b9ab14ee7f4394f0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.279824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ad0877d583d4550b9ab14ee7f4394f0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.291141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6251e057-95c1-4de3-b1a5-7cd5f06f4dad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ad0877d583d4550b9ab14ee7f4394f0'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.316554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1207/2401] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:15:24 2022] DELETE /identity/v3/users/0ad0877d583d4550b9ab14ee7f4394f0 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:15:24.500691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e359080a-ef21-48f3-aaba-795b376bff4a service swift] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:24.513126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e359080a-ef21-48f3-aaba-795b376bff4a service swift] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:24.557124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:24.592583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:24.593211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:24.593488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:24.593760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] PATH_INFO: `/v3/projects/1e90b31934484572bf7c7bdcbc0fdf2d` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:24.594863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e90b31934484572bf7c7bdcbc0fdf2d)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:24.597650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:24.651963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e90b31934484572bf7c7bdcbc0fdf2d', 'request_id': 'req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.660870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e90b31934484572bf7c7bdcbc0fdf2d', 'request_id': 'req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.666865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e90b31934484572bf7c7bdcbc0fdf2d', 'request_id': 'req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.679516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] The token cache is being invalidate because project 1e90b31934484572bf7c7bdcbc0fdf2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:15:24.679965 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.680715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5844baf9-4d18-491b-8d9c-f7f1ff586ad3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:24.682585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1195/2402] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:15:24 2022] DELETE /identity/v3/projects/1e90b31934484572bf7c7bdcbc0fdf2d => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:15:24.687204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1955b5-504a-49c7-82b1-b3ee29ee10bb None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:24.687921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1955b5-504a-49c7-82b1-b3ee29ee10bb None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:24.688564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1955b5-504a-49c7-82b1-b3ee29ee10bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:24.711266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5e1955b5-504a-49c7-82b1-b3ee29ee10bb None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:24.712544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5e1955b5-504a-49c7-82b1-b3ee29ee10bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:24.773506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1208/2403] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:24.782542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a788678-5d89-48a3-b758-892bb9caf392 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:24.782826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a788678-5d89-48a3-b758-892bb9caf392 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:24.783130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a788678-5d89-48a3-b758-892bb9caf392 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:24.802971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-5a788678-5d89-48a3-b758-892bb9caf392 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:24.804251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5a788678-5d89-48a3-b758-892bb9caf392 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:24.812472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1196/2404] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:24.817655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1955b5-504a-49c7-82b1-b3ee29ee10bb None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:24.822984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:24.833350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1955b5-504a-49c7-82b1-b3ee29ee10bb None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:24.838500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:24.839189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:24.839576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:24.839986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:24.840744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:15:24.841193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:24.842806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:24.855626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1209/2405] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:15:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:15:24.860269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a788678-5d89-48a3-b758-892bb9caf392 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:24.870082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a788678-5d89-48a3-b758-892bb9caf392 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:24.874810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c971b-f745-4a3a-9c09-c2cda43eb133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:24.875419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c971b-f745-4a3a-9c09-c2cda43eb133 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:24.875708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c971b-f745-4a3a-9c09-c2cda43eb133 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:24.880092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c971b-f745-4a3a-9c09-c2cda43eb133 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:24.880985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190c971b-f745-4a3a-9c09-c2cda43eb133 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:24.883104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190c971b-f745-4a3a-9c09-c2cda43eb133 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:24.913314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1197/2406] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:15:24 2022] POST /identity/v3/projects => generated 391 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:15:24.917199 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:24.928056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ae893c-58fc-482f-80bd-90efc3cbb57e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:24.991310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a223631-0952-4a40-bc8e-149da4248cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:24.991955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a223631-0952-4a40-bc8e-149da4248cb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:24.992239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a223631-0952-4a40-bc8e-149da4248cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:24.992515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a223631-0952-4a40-bc8e-149da4248cb2 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:24.993154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a223631-0952-4a40-bc8e-149da4248cb2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:24.995085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a223631-0952-4a40-bc8e-149da4248cb2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:25.016084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1210/2407] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:15:24 2022] POST /identity/v3/users => generated 296 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:15:25.021346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c971b-f745-4a3a-9c09-c2cda43eb133 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:25.031763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c971b-f745-4a3a-9c09-c2cda43eb133 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:25.037973 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c13d3ff-5a5b-4989-b854-cfed5d902089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:25.038642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c13d3ff-5a5b-4989-b854-cfed5d902089 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:25.039025 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c13d3ff-5a5b-4989-b854-cfed5d902089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:25.039286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c13d3ff-5a5b-4989-b854-cfed5d902089 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:25.040125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c13d3ff-5a5b-4989-b854-cfed5d902089 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:25.041767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c13d3ff-5a5b-4989-b854-cfed5d902089 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:25.053799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1198/2408] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:15:25.058194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a223631-0952-4a40-bc8e-149da4248cb2 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:25.069652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a223631-0952-4a40-bc8e-149da4248cb2 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:25.074315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1276100-9cf9-4af6-ab85-d11fbb66bcb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:25.074903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1276100-9cf9-4af6-ab85-d11fbb66bcb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:25.075185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1276100-9cf9-4af6-ab85-d11fbb66bcb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:25.075453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1276100-9cf9-4af6-ab85-d11fbb66bcb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:25.076127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1276100-9cf9-4af6-ab85-d11fbb66bcb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:25.077954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1276100-9cf9-4af6-ab85-d11fbb66bcb7 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:25.093050 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1211/2409] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:15:25.097999 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c13d3ff-5a5b-4989-b854-cfed5d902089 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:25.108348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c13d3ff-5a5b-4989-b854-cfed5d902089 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:25.113664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:25.114333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:25.114611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:25.114883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] PATH_INFO: `/v3/projects/fb8b6db01ffd494d8a4483bf707100dc/users/44a2f9b6c36e4010ae50ba5fb4e65199/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:25.130512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb8b6db01ffd494d8a4483bf707100dc, user_id=44a2f9b6c36e4010ae50ba5fb4e65199, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:25.133207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:25.148246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1199/2410] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:15:25 2022] PUT /identity/v3/projects/fb8b6db01ffd494d8a4483bf707100dc/users/44a2f9b6c36e4010ae50ba5fb4e65199/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:15:25.159024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e381cf8a-5ef4-4c2f-a776-d6f55082e16d None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:25.159513 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e381cf8a-5ef4-4c2f-a776-d6f55082e16d None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:25.159849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e381cf8a-5ef4-4c2f-a776-d6f55082e16d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:25.190106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e381cf8a-5ef4-4c2f-a776-d6f55082e16d None None] MFA Rules not processed for user `44a2f9b6c36e4010ae50ba5fb4e65199`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:25.191536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e381cf8a-5ef4-4c2f-a776-d6f55082e16d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:25.262957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1212/2411] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:15:25 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:25.270990 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:25.273014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:25.342287 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adf97cf-10fc-4b44-a805-5b01e1458a63 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:25.349308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774162c2-b035-47e7-a094-1adaaf17b013 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:25.349937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774162c2-b035-47e7-a094-1adaaf17b013 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:25.350224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774162c2-b035-47e7-a094-1adaaf17b013 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:25.350886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774162c2-b035-47e7-a094-1adaaf17b013 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:25.364746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774162c2-b035-47e7-a094-1adaaf17b013 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:25.367050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774162c2-b035-47e7-a094-1adaaf17b013 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:25.381429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1200/2412] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:25 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:35.256270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=71,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:35.374388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:50.635828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e381cf8a-5ef4-4c2f-a776-d6f55082e16d None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:50.637275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e381cf8a-5ef4-4c2f-a776-d6f55082e16d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:50.712963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e381cf8a-5ef4-4c2f-a776-d6f55082e16d None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:50.718408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284929a-e91b-473c-86c3-485a1d9e971f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:50.719006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284929a-e91b-473c-86c3-485a1d9e971f service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:50.719290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284929a-e91b-473c-86c3-485a1d9e971f service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:50.719568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284929a-e91b-473c-86c3-485a1d9e971f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:50.721326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4284929a-e91b-473c-86c3-485a1d9e971f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:50.733972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284929a-e91b-473c-86c3-485a1d9e971f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:50.735492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284929a-e91b-473c-86c3-485a1d9e971f service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:50.745755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1213/2413] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:51.957777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774162c2-b035-47e7-a094-1adaaf17b013 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:52.019322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774162c2-b035-47e7-a094-1adaaf17b013 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:52.023893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:52.024512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:52.024793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:52.025067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] PATH_INFO: `/v3/users/44a2f9b6c36e4010ae50ba5fb4e65199` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:52.026178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44a2f9b6c36e4010ae50ba5fb4e65199)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:52.028120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:52.074587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44a2f9b6c36e4010ae50ba5fb4e65199'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.081566 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44a2f9b6c36e4010ae50ba5fb4e65199'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.082181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44a2f9b6c36e4010ae50ba5fb4e65199'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.082817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44a2f9b6c36e4010ae50ba5fb4e65199'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.089831 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44a2f9b6c36e4010ae50ba5fb4e65199'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.107155 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1201/2414] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:15:51 2022] DELETE /identity/v3/users/44a2f9b6c36e4010ae50ba5fb4e65199 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:15:52.111624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f53c0a8-bdde-414e-ab9a-57aad9e01c13 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:52.112128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f53c0a8-bdde-414e-ab9a-57aad9e01c13 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:52.112259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f53c0a8-bdde-414e-ab9a-57aad9e01c13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:52.131328 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7f53c0a8-bdde-414e-ab9a-57aad9e01c13 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:52.132548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7f53c0a8-bdde-414e-ab9a-57aad9e01c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:52.191418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1214/2415] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:15:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:52.201636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:52.203041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:52.259521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f42e12-5d0f-4a08-b6cb-f9dc9a0dd152 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:52.263204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14629639-10b6-4397-bf4c-eb57a6d7c275 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:52.263834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14629639-10b6-4397-bf4c-eb57a6d7c275 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:52.264160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14629639-10b6-4397-bf4c-eb57a6d7c275 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:52.264434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14629639-10b6-4397-bf4c-eb57a6d7c275 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:52.273465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14629639-10b6-4397-bf4c-eb57a6d7c275 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:52.274780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14629639-10b6-4397-bf4c-eb57a6d7c275 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:52.284185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1202/2416] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:15:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:15:52.586346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f53c0a8-bdde-414e-ab9a-57aad9e01c13 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:52.587867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7f53c0a8-bdde-414e-ab9a-57aad9e01c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:52.600143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f53c0a8-bdde-414e-ab9a-57aad9e01c13 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:52.605438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:52.606073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:52.606357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:52.606635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] PATH_INFO: `/v3/projects/fb8b6db01ffd494d8a4483bf707100dc` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:52.607804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb8b6db01ffd494d8a4483bf707100dc)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:52.609723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:52.665928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb8b6db01ffd494d8a4483bf707100dc', 'request_id': 'req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.676967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb8b6db01ffd494d8a4483bf707100dc', 'request_id': 'req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.683424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb8b6db01ffd494d8a4483bf707100dc', 'request_id': 'req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.696101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] The token cache is being invalidate because project fb8b6db01ffd494d8a4483bf707100dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:15:52.696396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.697102 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:15:52.698528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1215/2417] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:15:52 2022] DELETE /identity/v3/projects/fb8b6db01ffd494d8a4483bf707100dc => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:15:53.897912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6041ffe-6649-41f9-a08f-493c5893aa12 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:53.898374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6041ffe-6649-41f9-a08f-493c5893aa12 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:53.898496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6041ffe-6649-41f9-a08f-493c5893aa12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:53.918089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f6041ffe-6649-41f9-a08f-493c5893aa12 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:15:53.922022 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f6041ffe-6649-41f9-a08f-493c5893aa12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:53.936034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:53.977643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1203/2418] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 15 14:15:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:15:53.999538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:15:54.000963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:15:54.012615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:15:54.072748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c19b3bd-1ce6-471b-acac-f3dbadc242eb admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:15:54.077787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:15:54.078388 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:15:54.078671 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:15:54.078946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:15:54.087763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:15:54.089356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:15:54.097390 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1216/2419] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:15:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:16:03.970315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=15 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:16:04.094221 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=28 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:16:13.029983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6041ffe-6649-41f9-a08f-493c5893aa12 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:16:13.031456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f6041ffe-6649-41f9-a08f-493c5893aa12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:16:13.104521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6041ffe-6649-41f9-a08f-493c5893aa12 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:16:13.109942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958248ea-964c-4d38-a88c-a1c9bef87ee6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:16:13.110542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958248ea-964c-4d38-a88c-a1c9bef87ee6 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:16:13.110820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958248ea-964c-4d38-a88c-a1c9bef87ee6 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:16:13.111093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958248ea-964c-4d38-a88c-a1c9bef87ee6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:16:13.112887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-958248ea-964c-4d38-a88c-a1c9bef87ee6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:16:13.171693 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958248ea-964c-4d38-a88c-a1c9bef87ee6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:16:13.173354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958248ea-964c-4d38-a88c-a1c9bef87ee6 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:16:13.182581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1204/2420] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:16:13 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:16:18.256022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:16:18.257594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:16:18.273283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb18d42e-15b9-41d7-abb3-9f5c0cd7b363 service swift] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:16:18.284162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:16:18.284774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:16:18.285055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:16:18.285349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:16:18.287283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:16:18.352876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:16:18.354587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:16:18.365490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1217/2421] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:16:18 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:16:23.178301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=56 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:16:28.359359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=28 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:16:36.719195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d67c5-dfdc-42ff-8954-0d5b009f1e62 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:16:36.719836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d67c5-dfdc-42ff-8954-0d5b009f1e62 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:16:36.719836 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d67c5-dfdc-42ff-8954-0d5b009f1e62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:16:36.740127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-758d67c5-dfdc-42ff-8954-0d5b009f1e62 None None] MFA Rules not processed for user `530e6587b39b4f4c98d7414b81453628`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:16:36.741414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-758d67c5-dfdc-42ff-8954-0d5b009f1e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:16:36.748322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1205/2422] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:16:36 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:16:41.933531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:16:41.947790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5658f505-c15e-4d24-acfc-e4a6879cce54 service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:16:41.955412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1ebcf5-7aa3-4886-bddf-9d6f9a6e596d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:16:41.955785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ebcf5-7aa3-4886-bddf-9d6f9a6e596d service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:16:41.956072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ebcf5-7aa3-4886-bddf-9d6f9a6e596d service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:16:41.956351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ebcf5-7aa3-4886-bddf-9d6f9a6e596d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:16:41.958771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2d1ebcf5-7aa3-4886-bddf-9d6f9a6e596d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:16:41.974941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ebcf5-7aa3-4886-bddf-9d6f9a6e596d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:16:41.976685 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ebcf5-7aa3-4886-bddf-9d6f9a6e596d service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:16:41.992788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1218/2423] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:16:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:16:46.734069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:16:51.983483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:17:01.104854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d67c5-dfdc-42ff-8954-0d5b009f1e62 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:17:01.119387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d67c5-dfdc-42ff-8954-0d5b009f1e62 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:17:01.125137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:17:01.125868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:17:01.126198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:17:01.126545 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:17:01.128524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:17:01.141809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:17:01.143574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:17:01.167902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1206/2424] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:17:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:17:05.078298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb7171b-9d22-4b86-b00a-70eac14b3316 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:17:05.078775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb7171b-9d22-4b86-b00a-70eac14b3316 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:17:05.078889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb7171b-9d22-4b86-b00a-70eac14b3316 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:17:05.097986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1cb7171b-9d22-4b86-b00a-70eac14b3316 None None] MFA Rules not processed for user `00374d3d3bf649749781792273e35e3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:17:05.099284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1cb7171b-9d22-4b86-b00a-70eac14b3316 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:17:05.152658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1219/2425] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:17:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:17:05.453240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:17:05.454713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:17:05.509923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e2ab49-6156-4d5f-b75f-bc740ce25bb8 service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:17:05.513526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:17:05.514120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:17:05.514402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:17:05.514677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:17:05.516463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:17:05.526267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:17:05.527695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:17:05.535515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1207/2426] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:17:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:17:15.146141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=25 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:17:15.532029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=32 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:17:51.071925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56950f71-7712-416a-8616-d533b542e0cf None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:17:51.072444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56950f71-7712-416a-8616-d533b542e0cf None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:17:51.072544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56950f71-7712-416a-8616-d533b542e0cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:17:51.102634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-56950f71-7712-416a-8616-d533b542e0cf None None] MFA Rules not processed for user `2049d66708ce47a794c14ad4c2040937`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:17:51.103887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-56950f71-7712-416a-8616-d533b542e0cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:17:51.128090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1220/2427] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:17:51 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:17:59.767841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:17:59.769248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:17:59.830793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416d74b4-b113-4e94-90b1-f8b57221b04c service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:17:59.835151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:17:59.835833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:17:59.836115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:17:59.836383 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:17:59.838629 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:17:59.938000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:17:59.939507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:17:59.964647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1208/2428] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:17:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:18:01.126346 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=15 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:03.546028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56950f71-7712-416a-8616-d533b542e0cf None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:03.548544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-56950f71-7712-416a-8616-d533b542e0cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:03.563486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56950f71-7712-416a-8616-d533b542e0cf None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:03.568039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:03.569452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:03.569727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:03.570188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] PATH_INFO: `/v3/users/530e6587b39b4f4c98d7414b81453628` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:03.571326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=530e6587b39b4f4c98d7414b81453628)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:03.573323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:03.620350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '530e6587b39b4f4c98d7414b81453628'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.628301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '530e6587b39b4f4c98d7414b81453628'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.628920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '530e6587b39b4f4c98d7414b81453628'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.629550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '530e6587b39b4f4c98d7414b81453628'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.637693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c2b7d14b-f874-4830-823a-b1b164c27f92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '530e6587b39b4f4c98d7414b81453628'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.656531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1221/2429] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:18:03 2022] DELETE /identity/v3/users/530e6587b39b4f4c98d7414b81453628 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:03.848294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:03.858747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e982f8a2-b924-43d2-a9ef-03dc0704d65b service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:03.918130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:03.918767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:03.919046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:03.919319 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] PATH_INFO: `/v3/projects/fab43b244e854c0296530db48bc1b975` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:03.920457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fab43b244e854c0296530db48bc1b975)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:03.922354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:03.934845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:03.972949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fab43b244e854c0296530db48bc1b975', 'request_id': 'req-59a8ae09-7cbe-422c-b5f8-472b3a45e617'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.980475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fab43b244e854c0296530db48bc1b975', 'request_id': 'req-59a8ae09-7cbe-422c-b5f8-472b3a45e617'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.986043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fab43b244e854c0296530db48bc1b975', 'request_id': 'req-59a8ae09-7cbe-422c-b5f8-472b3a45e617'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.998171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] The token cache is being invalidate because project fab43b244e854c0296530db48bc1b975 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:18:03.998486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:03.999139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-59a8ae09-7cbe-422c-b5f8-472b3a45e617 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:04.000648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1209/2430] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:18:03 2022] DELETE /identity/v3/projects/fab43b244e854c0296530db48bc1b975 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:04.006508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c16d4d-f769-4c37-994b-9804391f9738 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.006809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c16d4d-f769-4c37-994b-9804391f9738 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.007094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c16d4d-f769-4c37-994b-9804391f9738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.068765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-64c16d4d-f769-4c37-994b-9804391f9738 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:04.070506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-64c16d4d-f769-4c37-994b-9804391f9738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:04.129189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1222/2431] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:04.138111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32575ec2-a935-4a19-85b7-a27d1068cf87 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.138399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32575ec2-a935-4a19-85b7-a27d1068cf87 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.138677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32575ec2-a935-4a19-85b7-a27d1068cf87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.158047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-32575ec2-a935-4a19-85b7-a27d1068cf87 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:04.159262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-32575ec2-a935-4a19-85b7-a27d1068cf87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:04.168162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1210/2432] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:04.172407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c16d4d-f769-4c37-994b-9804391f9738 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.182860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c16d4d-f769-4c37-994b-9804391f9738 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.187420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.188121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.188466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.188823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.189571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:18:04.189934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.191624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.204526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1223/2433] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:18:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:18:04.209505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32575ec2-a935-4a19-85b7-a27d1068cf87 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.218875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32575ec2-a935-4a19-85b7-a27d1068cf87 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.223651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7191ef53-45a1-43b1-9ab8-ead2d3d6dedb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.224284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7191ef53-45a1-43b1-9ab8-ead2d3d6dedb admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.224561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7191ef53-45a1-43b1-9ab8-ead2d3d6dedb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.224830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7191ef53-45a1-43b1-9ab8-ead2d3d6dedb admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.226800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7191ef53-45a1-43b1-9ab8-ead2d3d6dedb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.228728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7191ef53-45a1-43b1-9ab8-ead2d3d6dedb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.256615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1211/2434] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:18:04 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:04.261138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.271753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83436a14-21ab-4713-9ad4-3fd17d7ea1b4 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.302609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:04.334757 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a92dc-2534-4ebc-ad8f-892a7a729831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.335502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a92dc-2534-4ebc-ad8f-892a7a729831 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.335818 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a92dc-2534-4ebc-ad8f-892a7a729831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.336096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361a92dc-2534-4ebc-ad8f-892a7a729831 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.336742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a92dc-2534-4ebc-ad8f-892a7a729831 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.338699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361a92dc-2534-4ebc-ad8f-892a7a729831 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.362001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1224/2435] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:18:04 2022] POST /identity/v3/users => generated 291 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:04.366786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7191ef53-45a1-43b1-9ab8-ead2d3d6dedb admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.377485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7191ef53-45a1-43b1-9ab8-ead2d3d6dedb admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.382800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2d5153-33fd-4e5a-9c6c-b3496e51beb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.383422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2d5153-33fd-4e5a-9c6c-b3496e51beb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.383732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2d5153-33fd-4e5a-9c6c-b3496e51beb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.384009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2d5153-33fd-4e5a-9c6c-b3496e51beb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.384665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2d5153-33fd-4e5a-9c6c-b3496e51beb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.386293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2d5153-33fd-4e5a-9c6c-b3496e51beb1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.398101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1212/2436] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:18:04.402312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a92dc-2534-4ebc-ad8f-892a7a729831 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.413156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361a92dc-2534-4ebc-ad8f-892a7a729831 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.418045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12a6dea-3dc2-427a-bf63-fcf801fcda4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.418909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a6dea-3dc2-427a-bf63-fcf801fcda4e admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.419242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a6dea-3dc2-427a-bf63-fcf801fcda4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.419521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a6dea-3dc2-427a-bf63-fcf801fcda4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.420219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a6dea-3dc2-427a-bf63-fcf801fcda4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.421867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a6dea-3dc2-427a-bf63-fcf801fcda4e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.436115 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1225/2437] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:04.437242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2d5153-33fd-4e5a-9c6c-b3496e51beb1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.448827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2d5153-33fd-4e5a-9c6c-b3496e51beb1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.453572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5bfe84-e2a0-445f-8b9b-e471b837025f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.454252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bfe84-e2a0-445f-8b9b-e471b837025f admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.454536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bfe84-e2a0-445f-8b9b-e471b837025f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.454814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bfe84-e2a0-445f-8b9b-e471b837025f admin admin] PATH_INFO: `/v3/projects/01e568301d4c491790d936be55a528e5/users/3bdda3bab6304a37b7298063470411ec/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.472708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5bfe84-e2a0-445f-8b9b-e471b837025f admin admin] RBAC: Authorizing `identity:create_grant(project_id=01e568301d4c491790d936be55a528e5, user_id=3bdda3bab6304a37b7298063470411ec, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.475715 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5bfe84-e2a0-445f-8b9b-e471b837025f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.489696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1213/2438] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:04 2022] PUT /identity/v3/projects/01e568301d4c491790d936be55a528e5/users/3bdda3bab6304a37b7298063470411ec/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:18:04.500668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e017a2-e99b-4f15-a07a-6867659915ef None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.501063 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e017a2-e99b-4f15-a07a-6867659915ef None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.501353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e017a2-e99b-4f15-a07a-6867659915ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.533181 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-00e017a2-e99b-4f15-a07a-6867659915ef None None] MFA Rules not processed for user `3bdda3bab6304a37b7298063470411ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:04.534448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-00e017a2-e99b-4f15-a07a-6867659915ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:04.602073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1226/2439] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:04 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:04.606600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5bfe84-e2a0-445f-8b9b-e471b837025f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.668783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5bfe84-e2a0-445f-8b9b-e471b837025f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.673390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3f2a3a-85f2-41ad-a561-a943ca3d679a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.673997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f2a3a-85f2-41ad-a561-a943ca3d679a admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.674283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f2a3a-85f2-41ad-a561-a943ca3d679a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.674554 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f2a3a-85f2-41ad-a561-a943ca3d679a admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.675189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3f2a3a-85f2-41ad-a561-a943ca3d679a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.677116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3f2a3a-85f2-41ad-a561-a943ca3d679a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.706555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1214/2440] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:18:04 2022] POST /identity/v3/projects => generated 381 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:04.710039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e017a2-e99b-4f15-a07a-6867659915ef None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.721290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e017a2-e99b-4f15-a07a-6867659915ef None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.798377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29981aa2-af6b-4451-8448-3f92761b81ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.799265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29981aa2-af6b-4451-8448-3f92761b81ee admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.799389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29981aa2-af6b-4451-8448-3f92761b81ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.799634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29981aa2-af6b-4451-8448-3f92761b81ee admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.800460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29981aa2-af6b-4451-8448-3f92761b81ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.802487 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29981aa2-af6b-4451-8448-3f92761b81ee admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.833002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1227/2441] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:18:04 2022] POST /identity/v3/users => generated 297 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:04.834214 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3f2a3a-85f2-41ad-a561-a943ca3d679a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.845872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3f2a3a-85f2-41ad-a561-a943ca3d679a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.851632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5a994-bb70-4039-a73f-ac5317675798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.852280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5a994-bb70-4039-a73f-ac5317675798 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.852527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5a994-bb70-4039-a73f-ac5317675798 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.852800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5a994-bb70-4039-a73f-ac5317675798 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.853444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f5a994-bb70-4039-a73f-ac5317675798 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.855076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f5a994-bb70-4039-a73f-ac5317675798 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.865938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1215/2442] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:04.870515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29981aa2-af6b-4451-8448-3f92761b81ee admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.880648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29981aa2-af6b-4451-8448-3f92761b81ee admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.885964 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401fb6c3-d8be-4e83-baa2-5fe64e49922c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.886610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401fb6c3-d8be-4e83-baa2-5fe64e49922c admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.886891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401fb6c3-d8be-4e83-baa2-5fe64e49922c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.887168 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401fb6c3-d8be-4e83-baa2-5fe64e49922c admin admin] PATH_INFO: `/v3/projects/57e958dc543e4e12a630d0c9f4c1a0df/users/8b6e4db0c974473c9cad018b1fa60e2b/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.903269 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401fb6c3-d8be-4e83-baa2-5fe64e49922c admin admin] RBAC: Authorizing `identity:create_grant(project_id=57e958dc543e4e12a630d0c9f4c1a0df, user_id=8b6e4db0c974473c9cad018b1fa60e2b, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.906049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401fb6c3-d8be-4e83-baa2-5fe64e49922c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:04.922512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1228/2443] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:04 2022] PUT /identity/v3/projects/57e958dc543e4e12a630d0c9f4c1a0df/users/8b6e4db0c974473c9cad018b1fa60e2b/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:04.923723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5a994-bb70-4039-a73f-ac5317675798 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:04.933923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5a994-bb70-4039-a73f-ac5317675798 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:04.941832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:04.995972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c84b6e6-2621-4360-9905-92b788d29621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:04.996487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c84b6e6-2621-4360-9905-92b788d29621 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:04.996764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c84b6e6-2621-4360-9905-92b788d29621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:04.997559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c84b6e6-2621-4360-9905-92b788d29621 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:04.997690 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c84b6e6-2621-4360-9905-92b788d29621 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:04.999237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c84b6e6-2621-4360-9905-92b788d29621 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:05.010267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1216/2444] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:04 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:18:05.015164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401fb6c3-d8be-4e83-baa2-5fe64e49922c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:05.020655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:05.027403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401fb6c3-d8be-4e83-baa2-5fe64e49922c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:05.032480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:05.033217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:05.033499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:05.033805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] PATH_INFO: `/v3/projects/57e958dc543e4e12a630d0c9f4c1a0df/users/8b6e4db0c974473c9cad018b1fa60e2b/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:05.035886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57e958dc543e4e12a630d0c9f4c1a0df, user_id=8b6e4db0c974473c9cad018b1fa60e2b, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:05.039570 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:05.050752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1229/2445] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:05 2022] PUT /identity/v3/projects/57e958dc543e4e12a630d0c9f4c1a0df/users/8b6e4db0c974473c9cad018b1fa60e2b/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:05.062145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847be7fc-58fb-4fc4-a341-dcadbf46d46d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:05.062432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847be7fc-58fb-4fc4-a341-dcadbf46d46d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:05.062728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847be7fc-58fb-4fc4-a341-dcadbf46d46d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:05.090599 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-847be7fc-58fb-4fc4-a341-dcadbf46d46d None None] MFA Rules not processed for user `8b6e4db0c974473c9cad018b1fa60e2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:05.091865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-847be7fc-58fb-4fc4-a341-dcadbf46d46d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:05.162428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1217/2446] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:05 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:05.174455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:05.176237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:05.238739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfebb7-a774-4284-acd1-e9fe7e7480f5 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:05.242656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044f5ab-88a7-48e3-adfd-1a17c474585a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:05.243272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044f5ab-88a7-48e3-adfd-1a17c474585a service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:05.243552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044f5ab-88a7-48e3-adfd-1a17c474585a service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:05.244282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044f5ab-88a7-48e3-adfd-1a17c474585a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:05.253545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6044f5ab-88a7-48e3-adfd-1a17c474585a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:05.256750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6044f5ab-88a7-48e3-adfd-1a17c474585a service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:05.267132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1230/2447] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:05 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:18:05.352335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847be7fc-58fb-4fc4-a341-dcadbf46d46d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:05.362185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847be7fc-58fb-4fc4-a341-dcadbf46d46d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:05.366838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0746be5c-f43d-4087-bdaa-97d3574ac327 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:05.367231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746be5c-f43d-4087-bdaa-97d3574ac327 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:05.367523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746be5c-f43d-4087-bdaa-97d3574ac327 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:05.367835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746be5c-f43d-4087-bdaa-97d3574ac327 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:05.434673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0746be5c-f43d-4087-bdaa-97d3574ac327 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:05.436693 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0746be5c-f43d-4087-bdaa-97d3574ac327 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:05.447548 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1218/2448] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:05 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:18:15.265699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:15.442930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=86 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:16.509583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044f5ab-88a7-48e3-adfd-1a17c474585a service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:16.583281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044f5ab-88a7-48e3-adfd-1a17c474585a service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:16.588963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:16.589605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:16.589882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:16.590157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] PATH_INFO: `/v3/users/3bdda3bab6304a37b7298063470411ec` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:16.591282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bdda3bab6304a37b7298063470411ec)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:16.593298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:16.654313 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bdda3bab6304a37b7298063470411ec'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:16.661315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bdda3bab6304a37b7298063470411ec'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:16.661976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bdda3bab6304a37b7298063470411ec'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:16.662716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bdda3bab6304a37b7298063470411ec'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:16.669637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bdda3bab6304a37b7298063470411ec'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:16.687033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1231/2449] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:18:16 2022] DELETE /identity/v3/users/3bdda3bab6304a37b7298063470411ec => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:16.691660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1921377-fc57-4306-ab2a-bd48b0a72a1c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:16.692141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1921377-fc57-4306-ab2a-bd48b0a72a1c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:16.692266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1921377-fc57-4306-ab2a-bd48b0a72a1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:16.712149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f1921377-fc57-4306-ab2a-bd48b0a72a1c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:16.713742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f1921377-fc57-4306-ab2a-bd48b0a72a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:16.773343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1219/2450] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:18:16.783169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:16.784610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:16.841294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15541255-1668-4091-9b78-a2dbf110aa7a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:16.844833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be92cf2b-3a7f-49d2-b27e-cf65df045c53 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:16.845428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be92cf2b-3a7f-49d2-b27e-cf65df045c53 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:16.845707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be92cf2b-3a7f-49d2-b27e-cf65df045c53 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:16.845979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be92cf2b-3a7f-49d2-b27e-cf65df045c53 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:16.855525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be92cf2b-3a7f-49d2-b27e-cf65df045c53 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:16.856988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be92cf2b-3a7f-49d2-b27e-cf65df045c53 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:16.867087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1232/2451] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:18:17.138359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1921377-fc57-4306-ab2a-bd48b0a72a1c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:17.139795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f1921377-fc57-4306-ab2a-bd48b0a72a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:17.151952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1921377-fc57-4306-ab2a-bd48b0a72a1c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:17.156985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:17.157615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:17.157891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:17.158161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] PATH_INFO: `/v3/projects/01e568301d4c491790d936be55a528e5` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:17.159678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01e568301d4c491790d936be55a528e5)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:17.161243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:17.210240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01e568301d4c491790d936be55a528e5', 'request_id': 'req-663e5929-d112-4d9c-826d-1e9e16815c42'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.217055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01e568301d4c491790d936be55a528e5', 'request_id': 'req-663e5929-d112-4d9c-826d-1e9e16815c42'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.222569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01e568301d4c491790d936be55a528e5', 'request_id': 'req-663e5929-d112-4d9c-826d-1e9e16815c42'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.234548 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] The token cache is being invalidate because project 01e568301d4c491790d936be55a528e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:18:17.234837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.235466 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.236948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1220/2452] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:18:17 2022] DELETE /identity/v3/projects/01e568301d4c491790d936be55a528e5 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:17.240939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be92cf2b-3a7f-49d2-b27e-cf65df045c53 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:17.250403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be92cf2b-3a7f-49d2-b27e-cf65df045c53 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:17.269780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:17.310229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:17.310888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:17.311169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:17.311448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] PATH_INFO: `/v3/users/8b6e4db0c974473c9cad018b1fa60e2b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:17.312601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b6e4db0c974473c9cad018b1fa60e2b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:17.314524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:17.360446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b6e4db0c974473c9cad018b1fa60e2b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.367868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b6e4db0c974473c9cad018b1fa60e2b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.368493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b6e4db0c974473c9cad018b1fa60e2b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.369109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b6e4db0c974473c9cad018b1fa60e2b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.376070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-41d6ade6-68ad-426d-864c-5a559d16fb65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b6e4db0c974473c9cad018b1fa60e2b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.393061 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1233/2453] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:18:17 2022] DELETE /identity/v3/users/8b6e4db0c974473c9cad018b1fa60e2b => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:17.670925 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:17.672484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:17.685947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663e5929-d112-4d9c-826d-1e9e16815c42 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:17.759448 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:17.760153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:17.760382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:17.760659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] PATH_INFO: `/v3/projects/57e958dc543e4e12a630d0c9f4c1a0df` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:17.761779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57e958dc543e4e12a630d0c9f4c1a0df)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:17.763679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:17.795328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:17.812986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57e958dc543e4e12a630d0c9f4c1a0df', 'request_id': 'req-9b9a5e9d-8316-4bb1-9019-b3dc97801927'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.821659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57e958dc543e4e12a630d0c9f4c1a0df', 'request_id': 'req-9b9a5e9d-8316-4bb1-9019-b3dc97801927'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.827357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57e958dc543e4e12a630d0c9f4c1a0df', 'request_id': 'req-9b9a5e9d-8316-4bb1-9019-b3dc97801927'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.839423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] The token cache is being invalidate because project 57e958dc543e4e12a630d0c9f4c1a0df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:18:17.839737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.840390 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9b9a5e9d-8316-4bb1-9019-b3dc97801927 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:17.841879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1221/2454] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:18:17 2022] DELETE /identity/v3/projects/57e958dc543e4e12a630d0c9f4c1a0df => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:17.846867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a8344-4fb0-40ad-872b-2a4967068203 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:17.847339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a8344-4fb0-40ad-872b-2a4967068203 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:17.847452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a8344-4fb0-40ad-872b-2a4967068203 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:17.868361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-470a8344-4fb0-40ad-872b-2a4967068203 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:17.869586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-470a8344-4fb0-40ad-872b-2a4967068203 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:17.932694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1234/2455] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:18:17.940589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a751a5e0-d44d-430e-aee9-4ee0a5177f49 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:17.940878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a751a5e0-d44d-430e-aee9-4ee0a5177f49 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:17.941157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a751a5e0-d44d-430e-aee9-4ee0a5177f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:17.966656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-a751a5e0-d44d-430e-aee9-4ee0a5177f49 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:17.967837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a751a5e0-d44d-430e-aee9-4ee0a5177f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:17.978059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1222/2456] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:17.982553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a8344-4fb0-40ad-872b-2a4967068203 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:17.994225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a8344-4fb0-40ad-872b-2a4967068203 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:17.999387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.000034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.000315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.000610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.001248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:18:18.001585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.003209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.019738 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1235/2457] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:18:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:18:18.022875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a751a5e0-d44d-430e-aee9-4ee0a5177f49 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.032743 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a751a5e0-d44d-430e-aee9-4ee0a5177f49 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.037677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7267e8-71f8-4e4c-bc1a-1b12e355119f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.038632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7267e8-71f8-4e4c-bc1a-1b12e355119f admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.038968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7267e8-71f8-4e4c-bc1a-1b12e355119f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.039333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7267e8-71f8-4e4c-bc1a-1b12e355119f admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.040101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7267e8-71f8-4e4c-bc1a-1b12e355119f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.042066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7267e8-71f8-4e4c-bc1a-1b12e355119f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.073323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1223/2458] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:18:18 2022] POST /identity/v3/projects => generated 375 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:18:18.077581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.088149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97535301-342a-432c-a45d-cd9f8e0d2737 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.114541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:18.151918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edd499-ef47-4b39-982c-d1179791356a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.152535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edd499-ef47-4b39-982c-d1179791356a admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.152826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edd499-ef47-4b39-982c-d1179791356a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.153103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edd499-ef47-4b39-982c-d1179791356a admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.153742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72edd499-ef47-4b39-982c-d1179791356a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.155712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72edd499-ef47-4b39-982c-d1179791356a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.178994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1236/2459] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:18:18 2022] POST /identity/v3/users => generated 288 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:18.183176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7267e8-71f8-4e4c-bc1a-1b12e355119f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.194694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7267e8-71f8-4e4c-bc1a-1b12e355119f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.201436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfef972-254f-43b5-8a54-fd7fda2f1649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.202233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfef972-254f-43b5-8a54-fd7fda2f1649 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.202521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfef972-254f-43b5-8a54-fd7fda2f1649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.202800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfef972-254f-43b5-8a54-fd7fda2f1649 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.203487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfef972-254f-43b5-8a54-fd7fda2f1649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.205094 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfef972-254f-43b5-8a54-fd7fda2f1649 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.216221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1224/2460] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:18.220466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edd499-ef47-4b39-982c-d1179791356a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.231825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edd499-ef47-4b39-982c-d1179791356a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.236541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9511e3f2-0d67-4511-9988-b5b9c6a7be02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.237152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9511e3f2-0d67-4511-9988-b5b9c6a7be02 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.237453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9511e3f2-0d67-4511-9988-b5b9c6a7be02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.237712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9511e3f2-0d67-4511-9988-b5b9c6a7be02 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.238368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9511e3f2-0d67-4511-9988-b5b9c6a7be02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.239996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9511e3f2-0d67-4511-9988-b5b9c6a7be02 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.257542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1237/2461] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:18.258502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfef972-254f-43b5-8a54-fd7fda2f1649 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.268623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfef972-254f-43b5-8a54-fd7fda2f1649 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.273424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a70833-ec95-4da3-a44d-e51e3516be4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.274165 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a70833-ec95-4da3-a44d-e51e3516be4d admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.274451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a70833-ec95-4da3-a44d-e51e3516be4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.274728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a70833-ec95-4da3-a44d-e51e3516be4d admin admin] PATH_INFO: `/v3/projects/689dbc84ca664f708caaea7aa00798b5/users/5b0a7fdc398e466ca761d80f3286c740/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.290359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a70833-ec95-4da3-a44d-e51e3516be4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=689dbc84ca664f708caaea7aa00798b5, user_id=5b0a7fdc398e466ca761d80f3286c740, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.292982 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a70833-ec95-4da3-a44d-e51e3516be4d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.311852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1225/2462] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:18 2022] PUT /identity/v3/projects/689dbc84ca664f708caaea7aa00798b5/users/5b0a7fdc398e466ca761d80f3286c740/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:18.323072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7093b6a-1e45-489a-ac6c-2aa765759715 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.323406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7093b6a-1e45-489a-ac6c-2aa765759715 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.323803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7093b6a-1e45-489a-ac6c-2aa765759715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.355092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-f7093b6a-1e45-489a-ac6c-2aa765759715 None None] MFA Rules not processed for user `5b0a7fdc398e466ca761d80f3286c740`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:18.356369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f7093b6a-1e45-489a-ac6c-2aa765759715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:18.426668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1238/2463] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:18.431276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a70833-ec95-4da3-a44d-e51e3516be4d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.493994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a70833-ec95-4da3-a44d-e51e3516be4d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.504348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48aa1e7-02e7-4ffa-bd87-eb8d583ca0c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.504978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48aa1e7-02e7-4ffa-bd87-eb8d583ca0c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.505260 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48aa1e7-02e7-4ffa-bd87-eb8d583ca0c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.505533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48aa1e7-02e7-4ffa-bd87-eb8d583ca0c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.506817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48aa1e7-02e7-4ffa-bd87-eb8d583ca0c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.508799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48aa1e7-02e7-4ffa-bd87-eb8d583ca0c1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.536134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1226/2464] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:18:18 2022] POST /identity/v3/projects => generated 375 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:18:18.540839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7093b6a-1e45-489a-ac6c-2aa765759715 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.555217 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7093b6a-1e45-489a-ac6c-2aa765759715 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.611460 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a9301-607c-4e55-87b2-69f4f97fa068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.612251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a9301-607c-4e55-87b2-69f4f97fa068 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.612541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a9301-607c-4e55-87b2-69f4f97fa068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.612816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a9301-607c-4e55-87b2-69f4f97fa068 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.613444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9a9301-607c-4e55-87b2-69f4f97fa068 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.615451 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9a9301-607c-4e55-87b2-69f4f97fa068 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.639415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1239/2465] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:18:18 2022] POST /identity/v3/users => generated 294 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:18.643323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48aa1e7-02e7-4ffa-bd87-eb8d583ca0c1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.656476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48aa1e7-02e7-4ffa-bd87-eb8d583ca0c1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.661862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04afd9b3-e77e-4226-8665-0ec1010517d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.662477 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04afd9b3-e77e-4226-8665-0ec1010517d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.662763 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04afd9b3-e77e-4226-8665-0ec1010517d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.663140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04afd9b3-e77e-4226-8665-0ec1010517d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.663892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04afd9b3-e77e-4226-8665-0ec1010517d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.665512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04afd9b3-e77e-4226-8665-0ec1010517d3 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.676492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1227/2466] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:18.680834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a9301-607c-4e55-87b2-69f4f97fa068 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.691261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a9301-607c-4e55-87b2-69f4f97fa068 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.697066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d665917b-c904-4b19-afda-bd5b228a39c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.697721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d665917b-c904-4b19-afda-bd5b228a39c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.698005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d665917b-c904-4b19-afda-bd5b228a39c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.698300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d665917b-c904-4b19-afda-bd5b228a39c7 admin admin] PATH_INFO: `/v3/projects/9d92a42b00be454cbaa9b7f7c0ce04d9/users/cc3da9341df54221a2a66c56796d78a3/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.714452 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d665917b-c904-4b19-afda-bd5b228a39c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d92a42b00be454cbaa9b7f7c0ce04d9, user_id=cc3da9341df54221a2a66c56796d78a3, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.717357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d665917b-c904-4b19-afda-bd5b228a39c7 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.730516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1240/2467] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:18 2022] PUT /identity/v3/projects/9d92a42b00be454cbaa9b7f7c0ce04d9/users/cc3da9341df54221a2a66c56796d78a3/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:18.734567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04afd9b3-e77e-4226-8665-0ec1010517d3 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.746467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04afd9b3-e77e-4226-8665-0ec1010517d3 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.767184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:18.802645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b4c18f-f5bc-4824-9071-7ed864c33f74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.803938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b4c18f-f5bc-4824-9071-7ed864c33f74 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.804218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b4c18f-f5bc-4824-9071-7ed864c33f74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.804486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b4c18f-f5bc-4824-9071-7ed864c33f74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.805115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b4c18f-f5bc-4824-9071-7ed864c33f74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.806695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b4c18f-f5bc-4824-9071-7ed864c33f74 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.816250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1228/2468] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:18 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:18.821468 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d665917b-c904-4b19-afda-bd5b228a39c7 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.826913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:18.832760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d665917b-c904-4b19-afda-bd5b228a39c7 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:18.838789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7330257e-bc01-41e6-80a3-bf0ac5fcbe95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:18.839447 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330257e-bc01-41e6-80a3-bf0ac5fcbe95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.839823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330257e-bc01-41e6-80a3-bf0ac5fcbe95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.840094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330257e-bc01-41e6-80a3-bf0ac5fcbe95 admin admin] PATH_INFO: `/v3/projects/9d92a42b00be454cbaa9b7f7c0ce04d9/users/cc3da9341df54221a2a66c56796d78a3/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.841986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7330257e-bc01-41e6-80a3-bf0ac5fcbe95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d92a42b00be454cbaa9b7f7c0ce04d9, user_id=cc3da9341df54221a2a66c56796d78a3, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:18.851203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7330257e-bc01-41e6-80a3-bf0ac5fcbe95 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:18.865236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1241/2469] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:18 2022] PUT /identity/v3/projects/9d92a42b00be454cbaa9b7f7c0ce04d9/users/cc3da9341df54221a2a66c56796d78a3/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:18.877201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d6ff88-7493-4a77-aa0b-efd0978a7880 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.877527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d6ff88-7493-4a77-aa0b-efd0978a7880 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.877804 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d6ff88-7493-4a77-aa0b-efd0978a7880 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.906636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-25d6ff88-7493-4a77-aa0b-efd0978a7880 None None] MFA Rules not processed for user `cc3da9341df54221a2a66c56796d78a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:18.907893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-25d6ff88-7493-4a77-aa0b-efd0978a7880 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:18.979003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1229/2470] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:18 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:18.982555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31844b0e-5b35-434a-85fb-d91336abb340 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:18.982840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31844b0e-5b35-434a-85fb-d91336abb340 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:18.983116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31844b0e-5b35-434a-85fb-d91336abb340 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:18.990208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d6ff88-7493-4a77-aa0b-efd0978a7880 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:18.992189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-25d6ff88-7493-4a77-aa0b-efd0978a7880 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:19.013488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-31844b0e-5b35-434a-85fb-d91336abb340 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:19.014731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-31844b0e-5b35-434a-85fb-d91336abb340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:19.078749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d6ff88-7493-4a77-aa0b-efd0978a7880 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:19.082355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1242/2471] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 15 14:18:18 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:18:19.082569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:19.083177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:19.083458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:19.083759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:19.126113 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:19.132240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31844b0e-5b35-434a-85fb-d91336abb340 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:19.133758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-31844b0e-5b35-434a-85fb-d91336abb340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:19.144277 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:19.145718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:19.155279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1230/2472] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:18 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:18:19.205152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31844b0e-5b35-434a-85fb-d91336abb340 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:19.210396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:19.211007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:19.211290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:19.211567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:19.219626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:19.221274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:19.230163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1243/2473] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:18:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 15 14:18:22.682822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:22.684212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:22.759954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:22.762982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:22.805899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f125f02b-9b43-4acf-9f44-64556945c67b service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:22.812807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:22.813422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:22.813705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:22.813980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:22.816084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:22.831450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:22.833306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:22.839442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cc9500-95cc-4ad1-b17d-ed26336b8a4c service swift] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:22.843326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:22.843966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:22.844294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:22.844522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:22.846969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:22.847939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1231/2474] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:18:22.872035 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:22.918299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:22.919787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:22.930131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1244/2475] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:18:25.913963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:25.924701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7177dfc7-0ca7-49b2-8a0a-23f8fa7b7c48 service placement] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:25.928762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e48ffe7-4946-4f07-b490-976b0a3613b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:25.929332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e48ffe7-4946-4f07-b490-976b0a3613b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:25.929617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e48ffe7-4946-4f07-b490-976b0a3613b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:25.929992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e48ffe7-4946-4f07-b490-976b0a3613b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:25.947200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e48ffe7-4946-4f07-b490-976b0a3613b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:25.948830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e48ffe7-4946-4f07-b490-976b0a3613b4 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:25.960859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1232/2476] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:25 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:18:26.102113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:26.103673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:26.117353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcda159-2229-4518-bb1f-843a939cbe13 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:26.122175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:26.122807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:26.123089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:26.123363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] PATH_INFO: `/v3/users/2bf8e22b8bb2459b83d1c065c0da7145` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:26.124532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bf8e22b8bb2459b83d1c065c0da7145)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:26.126462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:26.183469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bf8e22b8bb2459b83d1c065c0da7145'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.190710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bf8e22b8bb2459b83d1c065c0da7145'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.191317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bf8e22b8bb2459b83d1c065c0da7145'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.191950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bf8e22b8bb2459b83d1c065c0da7145'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.199441 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bf8e22b8bb2459b83d1c065c0da7145'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.222634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1245/2477] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:18:26 2022] DELETE /identity/v3/users/2bf8e22b8bb2459b83d1c065c0da7145 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:26.413018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e48ffe7-4946-4f07-b490-976b0a3613b4 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:26.428560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e48ffe7-4946-4f07-b490-976b0a3613b4 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:26.485323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:26.485946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:26.486226 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:26.486499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] PATH_INFO: `/v3/projects/2d56d6c8fce14f2f9fe424a5d0ae2046` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:26.487649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d56d6c8fce14f2f9fe424a5d0ae2046)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:26.489632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:26.530232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:26.538676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d56d6c8fce14f2f9fe424a5d0ae2046', 'request_id': 'req-4a7fd7e4-2971-4f73-b115-2259affa3979'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.545499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d56d6c8fce14f2f9fe424a5d0ae2046', 'request_id': 'req-4a7fd7e4-2971-4f73-b115-2259affa3979'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.551147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d56d6c8fce14f2f9fe424a5d0ae2046', 'request_id': 'req-4a7fd7e4-2971-4f73-b115-2259affa3979'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.562851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] The token cache is being invalidate because project 2d56d6c8fce14f2f9fe424a5d0ae2046 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:18:26.563148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.563817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.565561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1233/2478] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:18:26 2022] DELETE /identity/v3/projects/2d56d6c8fce14f2f9fe424a5d0ae2046 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:18:26.570188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:26.571600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:26.584652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6432ed0e-e15f-4768-9876-e343a12680ad admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:26.640791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:26.641426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:26.641712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:26.641988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] PATH_INFO: `/v3/users/88b6200f2c7f4236bdbcfe63a6fcff4c` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:26.643114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88b6200f2c7f4236bdbcfe63a6fcff4c)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:26.645093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:26.680038 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:26.693684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88b6200f2c7f4236bdbcfe63a6fcff4c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.700720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88b6200f2c7f4236bdbcfe63a6fcff4c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.701342 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88b6200f2c7f4236bdbcfe63a6fcff4c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.701947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88b6200f2c7f4236bdbcfe63a6fcff4c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.708797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88b6200f2c7f4236bdbcfe63a6fcff4c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:26.726277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1246/2479] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:18:26 2022] DELETE /identity/v3/users/88b6200f2c7f4236bdbcfe63a6fcff4c => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:26.900459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:26.902047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:26.915424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7fd7e4-2971-4f73-b115-2259affa3979 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:26.977857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:26.978489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:26.978767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:26.979039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] PATH_INFO: `/v3/projects/7c1acf0de9e5442ca8a9a310de0238ed` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:26.980188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c1acf0de9e5442ca8a9a310de0238ed)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:26.982093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:27.027568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c1acf0de9e5442ca8a9a310de0238ed', 'request_id': 'req-b00a2787-055e-41a6-aa00-1fbf766d65c2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:27.034034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c1acf0de9e5442ca8a9a310de0238ed', 'request_id': 'req-b00a2787-055e-41a6-aa00-1fbf766d65c2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:27.040680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c1acf0de9e5442ca8a9a310de0238ed', 'request_id': 'req-b00a2787-055e-41a6-aa00-1fbf766d65c2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:27.052682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] The token cache is being invalidate because project 7c1acf0de9e5442ca8a9a310de0238ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:18:27.053122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:27.054084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:27.056053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1234/2480] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:18:26 2022] DELETE /identity/v3/projects/7c1acf0de9e5442ca8a9a310de0238ed => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:18:36.722232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=17,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:37.048765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=61,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:50.988718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:50.990140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:51.059915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e7344-e3cf-4dc1-b0ee-4b18ddb39ca8 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:51.117545 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72c18ad-20ad-4a1a-b13a-599c829fec07 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:51.118144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c18ad-20ad-4a1a-b13a-599c829fec07 service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:51.118420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c18ad-20ad-4a1a-b13a-599c829fec07 service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:51.118698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c18ad-20ad-4a1a-b13a-599c829fec07 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:51.121242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c72c18ad-20ad-4a1a-b13a-599c829fec07 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:51.132267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72c18ad-20ad-4a1a-b13a-599c829fec07 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:51.133846 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72c18ad-20ad-4a1a-b13a-599c829fec07 service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:51.143652 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1247/2481] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:18:56.004521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:56.005901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:56.068740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00a2787-055e-41a6-aa00-1fbf766d65c2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:56.074604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:56.075257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:56.075535 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:56.075838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] PATH_INFO: `/v3/users/5b0a7fdc398e466ca761d80f3286c740` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:56.076960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b0a7fdc398e466ca761d80f3286c740)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:56.079025 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:56.124025 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b0a7fdc398e466ca761d80f3286c740'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.130794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b0a7fdc398e466ca761d80f3286c740'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.131396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b0a7fdc398e466ca761d80f3286c740'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.131996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b0a7fdc398e466ca761d80f3286c740'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.139832 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b0a7fdc398e466ca761d80f3286c740'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.156871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1235/2482] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:18:56 2022] DELETE /identity/v3/users/5b0a7fdc398e466ca761d80f3286c740 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:56.161280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a23e73-7197-4d49-a6ff-2cb3fbc74709 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:56.161760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a23e73-7197-4d49-a6ff-2cb3fbc74709 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:56.161858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a23e73-7197-4d49-a6ff-2cb3fbc74709 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:56.181384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a3a23e73-7197-4d49-a6ff-2cb3fbc74709 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:56.183203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a3a23e73-7197-4d49-a6ff-2cb3fbc74709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:56.241031 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1248/2483] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:56.251306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:56.252712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:56.308680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39d1f5d-bd63-441d-9fa0-eb415101e5a0 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:56.312338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b25b4-afb6-461a-83c7-844cf632dc4a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:56.312943 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b25b4-afb6-461a-83c7-844cf632dc4a service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:56.313221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b25b4-afb6-461a-83c7-844cf632dc4a service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:56.313494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b25b4-afb6-461a-83c7-844cf632dc4a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:56.322935 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078b25b4-afb6-461a-83c7-844cf632dc4a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:56.324884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078b25b4-afb6-461a-83c7-844cf632dc4a service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:56.334151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1236/2484] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:18:56.597023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a23e73-7197-4d49-a6ff-2cb3fbc74709 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:56.598492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a3a23e73-7197-4d49-a6ff-2cb3fbc74709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:56.611584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a23e73-7197-4d49-a6ff-2cb3fbc74709 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:56.616422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:56.617084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:56.617357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:56.617628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] PATH_INFO: `/v3/projects/689dbc84ca664f708caaea7aa00798b5` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:56.618774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] RBAC: Authorizing `identity:delete_project(project_id=689dbc84ca664f708caaea7aa00798b5)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:56.621185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:56.654236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:56.672610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '689dbc84ca664f708caaea7aa00798b5', 'request_id': 'req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.679562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '689dbc84ca664f708caaea7aa00798b5', 'request_id': 'req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.685282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '689dbc84ca664f708caaea7aa00798b5', 'request_id': 'req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.696732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] The token cache is being invalidate because project 689dbc84ca664f708caaea7aa00798b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:18:56.697020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.697655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.699142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1249/2485] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:18:56 2022] DELETE /identity/v3/projects/689dbc84ca664f708caaea7aa00798b5 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:56.703114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b25b4-afb6-461a-83c7-844cf632dc4a service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:56.712589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b25b4-afb6-461a-83c7-844cf632dc4a service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:56.731318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:56.773106 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:56.773651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:56.773930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:56.774203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] PATH_INFO: `/v3/users/cc3da9341df54221a2a66c56796d78a3` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:56.775315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc3da9341df54221a2a66c56796d78a3)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:56.777475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:56.822000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc3da9341df54221a2a66c56796d78a3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.830092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc3da9341df54221a2a66c56796d78a3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.830792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc3da9341df54221a2a66c56796d78a3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.831465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc3da9341df54221a2a66c56796d78a3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.838695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-0125c002-4ef7-434d-811e-7857b1f6baa5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc3da9341df54221a2a66c56796d78a3'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:56.855700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1237/2486] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:18:56 2022] DELETE /identity/v3/users/cc3da9341df54221a2a66c56796d78a3 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:57.125046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:57.126503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:57.139806 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06b05b5-9c5c-4afa-a1ee-ff82a215535e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:57.196787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:57.197476 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.197692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.197965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] PATH_INFO: `/v3/projects/9d92a42b00be454cbaa9b7f7c0ce04d9` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.199308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d92a42b00be454cbaa9b7f7c0ce04d9)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:57.201266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:57.247520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d92a42b00be454cbaa9b7f7c0ce04d9', 'request_id': 'req-977a6806-d1c0-414b-81a6-0485b09c8339'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:57.254686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d92a42b00be454cbaa9b7f7c0ce04d9', 'request_id': 'req-977a6806-d1c0-414b-81a6-0485b09c8339'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:57.260372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d92a42b00be454cbaa9b7f7c0ce04d9', 'request_id': 'req-977a6806-d1c0-414b-81a6-0485b09c8339'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:57.271343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] The token cache is being invalidate because project 9d92a42b00be454cbaa9b7f7c0ce04d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:18:57.271636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:57.272366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-977a6806-d1c0-414b-81a6-0485b09c8339 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:18:57.273845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1250/2487] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:18:57 2022] DELETE /identity/v3/projects/9d92a42b00be454cbaa9b7f7c0ce04d9 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:57.279238 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d5e357-0351-4a80-bf45-6b1a8dfae151 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.279751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d5e357-0351-4a80-bf45-6b1a8dfae151 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.279867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d5e357-0351-4a80-bf45-6b1a8dfae151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.299634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-72d5e357-0351-4a80-bf45-6b1a8dfae151 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:57.300920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-72d5e357-0351-4a80-bf45-6b1a8dfae151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:57.359615 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1238/2488] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:57.367439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8129b629-53ff-45a5-9b84-edfb65c335dd None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.367578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8129b629-53ff-45a5-9b84-edfb65c335dd None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.367915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8129b629-53ff-45a5-9b84-edfb65c335dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.387974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8129b629-53ff-45a5-9b84-edfb65c335dd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:57.389194 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8129b629-53ff-45a5-9b84-edfb65c335dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:57.397384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1251/2489] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:57.402161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d5e357-0351-4a80-bf45-6b1a8dfae151 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:57.411725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d5e357-0351-4a80-bf45-6b1a8dfae151 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:57.416234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:57.416873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.417152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.417429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.418066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:18:57.418369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:57.419982 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:57.431162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1239/2490] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:18:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:18:57.435966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8129b629-53ff-45a5-9b84-edfb65c335dd None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:57.445461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8129b629-53ff-45a5-9b84-edfb65c335dd None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:57.450834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47368aa9-0910-45bf-9260-9407feac9515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:57.451425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47368aa9-0910-45bf-9260-9407feac9515 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.451742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47368aa9-0910-45bf-9260-9407feac9515 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.452039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47368aa9-0910-45bf-9260-9407feac9515 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.452664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47368aa9-0910-45bf-9260-9407feac9515 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:57.454588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47368aa9-0910-45bf-9260-9407feac9515 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:57.481251 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1252/2491] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:18:57 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:57.485311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:57.495745 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6438288-ba40-47d9-b286-90cd0a901f9a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:57.522579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:57.554455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb4e5e8-8a11-4ea7-8e5d-50c73045872d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:57.554976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4e5e8-8a11-4ea7-8e5d-50c73045872d admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.555301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4e5e8-8a11-4ea7-8e5d-50c73045872d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.555575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4e5e8-8a11-4ea7-8e5d-50c73045872d admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.556242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb4e5e8-8a11-4ea7-8e5d-50c73045872d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:57.559044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb4e5e8-8a11-4ea7-8e5d-50c73045872d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:57.579960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1240/2492] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:18:57 2022] POST /identity/v3/users => generated 287 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:57.584043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47368aa9-0910-45bf-9260-9407feac9515 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:57.594329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47368aa9-0910-45bf-9260-9407feac9515 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:57.598956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f7fc90-472a-44dd-9e77-05ce44387108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:57.599539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7fc90-472a-44dd-9e77-05ce44387108 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.599849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7fc90-472a-44dd-9e77-05ce44387108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.600120 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7fc90-472a-44dd-9e77-05ce44387108 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.600752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f7fc90-472a-44dd-9e77-05ce44387108 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:57.602338 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f7fc90-472a-44dd-9e77-05ce44387108 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:57.612339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1253/2493] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:57.616162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb4e5e8-8a11-4ea7-8e5d-50c73045872d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:57.627061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb4e5e8-8a11-4ea7-8e5d-50c73045872d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:57.631674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702d712c-4f0d-450e-9c56-6fb56b8bdb8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:57.632300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d712c-4f0d-450e-9c56-6fb56b8bdb8c admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.632582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d712c-4f0d-450e-9c56-6fb56b8bdb8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.632857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d712c-4f0d-450e-9c56-6fb56b8bdb8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.633497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d712c-4f0d-450e-9c56-6fb56b8bdb8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:57.635086 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d712c-4f0d-450e-9c56-6fb56b8bdb8c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:57.646492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1241/2494] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:57.650569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f7fc90-472a-44dd-9e77-05ce44387108 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:57.660789 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f7fc90-472a-44dd-9e77-05ce44387108 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:57.665349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:57.665988 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.666321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.666686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] PATH_INFO: `/v3/projects/317a725ec099477ba549f7b51773419e/users/95a08be34e8a48b59086f12eee055e01/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.669556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:57.684686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=317a725ec099477ba549f7b51773419e, user_id=95a08be34e8a48b59086f12eee055e01, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:57.687454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:57.700900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1254/2495] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:57 2022] PUT /identity/v3/projects/317a725ec099477ba549f7b51773419e/users/95a08be34e8a48b59086f12eee055e01/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:57.705825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c074a23-c9ef-45b8-9022-c3e11cf7f443 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.706110 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c074a23-c9ef-45b8-9022-c3e11cf7f443 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.706386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c074a23-c9ef-45b8-9022-c3e11cf7f443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.725623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-1c074a23-c9ef-45b8-9022-c3e11cf7f443 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:57.726827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1c074a23-c9ef-45b8-9022-c3e11cf7f443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:57.785400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1242/2496] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:57.795698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:57.797387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:57.854037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee7bcba-73f7-46af-9537-26d0d06f8e7d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:57.857612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db042709-a0f3-4e95-8e5e-89f68c036224 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:57.858208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db042709-a0f3-4e95-8e5e-89f68c036224 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:57.858490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db042709-a0f3-4e95-8e5e-89f68c036224 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:57.858763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db042709-a0f3-4e95-8e5e-89f68c036224 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:57.867653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db042709-a0f3-4e95-8e5e-89f68c036224 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:57.869113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db042709-a0f3-4e95-8e5e-89f68c036224 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:57.878389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1255/2497] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:18:58.521693 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f2aa8e-7bad-4540-a373-8d681026b23a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:58.522246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f2aa8e-7bad-4540-a373-8d681026b23a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:58.522246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f2aa8e-7bad-4540-a373-8d681026b23a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:58.551229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-36f2aa8e-7bad-4540-a373-8d681026b23a None None] MFA Rules not processed for user `95a08be34e8a48b59086f12eee055e01`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:58.552548 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-36f2aa8e-7bad-4540-a373-8d681026b23a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:58.621042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1243/2498] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:58.625241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db042709-a0f3-4e95-8e5e-89f68c036224 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:58.634968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db042709-a0f3-4e95-8e5e-89f68c036224 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:58.639974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f72775d-2b5e-4c2b-bea9-15f73b9e86aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:58.640587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f72775d-2b5e-4c2b-bea9-15f73b9e86aa admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:58.640871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f72775d-2b5e-4c2b-bea9-15f73b9e86aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:58.641182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f72775d-2b5e-4c2b-bea9-15f73b9e86aa admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:58.641831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f72775d-2b5e-4c2b-bea9-15f73b9e86aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:58.643920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f72775d-2b5e-4c2b-bea9-15f73b9e86aa admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:58.672975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1256/2499] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:18:58 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:18:58.677030 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f2aa8e-7bad-4540-a373-8d681026b23a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:58.688863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f2aa8e-7bad-4540-a373-8d681026b23a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:58.722385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:58.747903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aabc250-b659-4233-8b81-09a7ca8723d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:58.748518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aabc250-b659-4233-8b81-09a7ca8723d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:58.748799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aabc250-b659-4233-8b81-09a7ca8723d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:58.749087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aabc250-b659-4233-8b81-09a7ca8723d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:58.749711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aabc250-b659-4233-8b81-09a7ca8723d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:58.751637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aabc250-b659-4233-8b81-09a7ca8723d6 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:58.772896 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1244/2500] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:18:58 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:18:58.777506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f72775d-2b5e-4c2b-bea9-15f73b9e86aa admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:58.787873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f72775d-2b5e-4c2b-bea9-15f73b9e86aa admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:58.793305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8904826-3d6c-40c1-b38a-255ff5a89bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:58.793905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8904826-3d6c-40c1-b38a-255ff5a89bed admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:58.794189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8904826-3d6c-40c1-b38a-255ff5a89bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:58.794469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8904826-3d6c-40c1-b38a-255ff5a89bed admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:58.795139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8904826-3d6c-40c1-b38a-255ff5a89bed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:58.796860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8904826-3d6c-40c1-b38a-255ff5a89bed admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:58.808538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1257/2501] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:58.812589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aabc250-b659-4233-8b81-09a7ca8723d6 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:58.822650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aabc250-b659-4233-8b81-09a7ca8723d6 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:58.827290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf99150-6798-48d6-9db3-882004de3431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:58.827983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf99150-6798-48d6-9db3-882004de3431 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:58.828250 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf99150-6798-48d6-9db3-882004de3431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:58.828532 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf99150-6798-48d6-9db3-882004de3431 admin admin] PATH_INFO: `/v3/projects/3bc7a35bc59041c3bed1c8bbea6f6bc2/users/e55a7549cdcf4adfb8b3a9de4aeeb8e6/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:58.843513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf99150-6798-48d6-9db3-882004de3431 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3bc7a35bc59041c3bed1c8bbea6f6bc2, user_id=e55a7549cdcf4adfb8b3a9de4aeeb8e6, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:58.846251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf99150-6798-48d6-9db3-882004de3431 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:58.860184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1245/2502] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:58 2022] PUT /identity/v3/projects/3bc7a35bc59041c3bed1c8bbea6f6bc2/users/e55a7549cdcf4adfb8b3a9de4aeeb8e6/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:58.864124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8904826-3d6c-40c1-b38a-255ff5a89bed admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:58.874186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8904826-3d6c-40c1-b38a-255ff5a89bed admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:58.928974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9581f0fe-89ab-4e62-b8db-2fc2f35c3e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:58.929496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9581f0fe-89ab-4e62-b8db-2fc2f35c3e3a admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:58.929771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9581f0fe-89ab-4e62-b8db-2fc2f35c3e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:58.930040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9581f0fe-89ab-4e62-b8db-2fc2f35c3e3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:58.930662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9581f0fe-89ab-4e62-b8db-2fc2f35c3e3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:58.932264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9581f0fe-89ab-4e62-b8db-2fc2f35c3e3a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:58.941950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1258/2503] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:18:58 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:18:58.946257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf99150-6798-48d6-9db3-882004de3431 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:58.956504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf99150-6798-48d6-9db3-882004de3431 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:58.961132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:58.961781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:58.962061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:58.962338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] PATH_INFO: `/v3/projects/3bc7a35bc59041c3bed1c8bbea6f6bc2/users/e55a7549cdcf4adfb8b3a9de4aeeb8e6/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:58.964820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3bc7a35bc59041c3bed1c8bbea6f6bc2, user_id=e55a7549cdcf4adfb8b3a9de4aeeb8e6, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:58.967457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:58.977664 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1246/2504] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:18:58 2022] PUT /identity/v3/projects/3bc7a35bc59041c3bed1c8bbea6f6bc2/users/e55a7549cdcf4adfb8b3a9de4aeeb8e6/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:18:59.595015 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fb393c-49df-4ad2-8672-4634433dbfb3 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:59.595362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fb393c-49df-4ad2-8672-4634433dbfb3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:59.595747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fb393c-49df-4ad2-8672-4634433dbfb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:59.624571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e6fb393c-49df-4ad2-8672-4634433dbfb3 None None] MFA Rules not processed for user `e55a7549cdcf4adfb8b3a9de4aeeb8e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:18:59.625868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e6fb393c-49df-4ad2-8672-4634433dbfb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:59.664101 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:18:59.696865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1259/2505] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:18:59 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:18:59.709174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:18:59.711029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:18:59.774033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a89198a-a820-4944-8d4d-395ba4ecdd64 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:18:59.777704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d085f50e-e0b3-4204-9b60-63e297f7ff8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:18:59.778301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085f50e-e0b3-4204-9b60-63e297f7ff8a service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:18:59.778583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085f50e-e0b3-4204-9b60-63e297f7ff8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:18:59.778863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085f50e-e0b3-4204-9b60-63e297f7ff8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:18:59.836192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d085f50e-e0b3-4204-9b60-63e297f7ff8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:18:59.837652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d085f50e-e0b3-4204-9b60-63e297f7ff8a service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:18:59.846650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1247/2506] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:18:59 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:01.213711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fb393c-49df-4ad2-8672-4634433dbfb3 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:01.215469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e6fb393c-49df-4ad2-8672-4634433dbfb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:01.230968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fb393c-49df-4ad2-8672-4634433dbfb3 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:01.237173 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f27ada-5dee-4353-972b-c4152004858b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:01.237791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f27ada-5dee-4353-972b-c4152004858b service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:01.238070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f27ada-5dee-4353-972b-c4152004858b service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:01.238340 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f27ada-5dee-4353-972b-c4152004858b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:01.250290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f27ada-5dee-4353-972b-c4152004858b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:01.251770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f27ada-5dee-4353-972b-c4152004858b service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:01.265565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1260/2507] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:01 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:09.842749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=75,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:11.259277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=18 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:14.383173 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d085f50e-e0b3-4204-9b60-63e297f7ff8a service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:14.446490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d085f50e-e0b3-4204-9b60-63e297f7ff8a service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:14.450911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:14.451529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:14.451839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:14.452109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] PATH_INFO: `/v3/users/95a08be34e8a48b59086f12eee055e01` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:14.454117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] RBAC: Authorizing `identity:delete_user(user_id=95a08be34e8a48b59086f12eee055e01)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:14.456130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:14.502531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95a08be34e8a48b59086f12eee055e01'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.509939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95a08be34e8a48b59086f12eee055e01'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.510491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95a08be34e8a48b59086f12eee055e01'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.511100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95a08be34e8a48b59086f12eee055e01'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.518933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95a08be34e8a48b59086f12eee055e01'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.536947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1248/2508] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:19:14 2022] DELETE /identity/v3/users/95a08be34e8a48b59086f12eee055e01 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:14.704508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f27ada-5dee-4353-972b-c4152004858b service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:14.720504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f27ada-5dee-4353-972b-c4152004858b service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:14.778032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:14.778664 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:14.778948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:14.779226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] PATH_INFO: `/v3/projects/317a725ec099477ba549f7b51773419e` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:14.780857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=317a725ec099477ba549f7b51773419e)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:14.782906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:14.829360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '317a725ec099477ba549f7b51773419e', 'request_id': 'req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.836571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '317a725ec099477ba549f7b51773419e', 'request_id': 'req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.842319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '317a725ec099477ba549f7b51773419e', 'request_id': 'req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.854116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] The token cache is being invalidate because project 317a725ec099477ba549f7b51773419e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:19:14.854410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.855064 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.856603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1261/2509] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:19:14 2022] DELETE /identity/v3/projects/317a725ec099477ba549f7b51773419e => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:14.860739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:14.862128 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:14.874875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792906fc-6200-44fd-b635-e5f8d1ea9bda admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:14.931751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:14.932393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:14.932702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:14.933030 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] PATH_INFO: `/v3/users/e55a7549cdcf4adfb8b3a9de4aeeb8e6` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:14.934181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] RBAC: Authorizing `identity:delete_user(user_id=e55a7549cdcf4adfb8b3a9de4aeeb8e6)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:14.936217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:14.960623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:14.989778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e55a7549cdcf4adfb8b3a9de4aeeb8e6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.997449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e55a7549cdcf4adfb8b3a9de4aeeb8e6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.998090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e55a7549cdcf4adfb8b3a9de4aeeb8e6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:14.998879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e55a7549cdcf4adfb8b3a9de4aeeb8e6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:15.010066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d0288b1d-a1be-40e3-994d-c223f90f4eef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e55a7549cdcf4adfb8b3a9de4aeeb8e6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:15.027992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1249/2510] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:19:14 2022] DELETE /identity/v3/users/e55a7549cdcf4adfb8b3a9de4aeeb8e6 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:15.198419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:15.199919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:15.215434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e0e4d4-ba0b-46be-82b0-1a26e7a3a7fa admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:15.277227 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:15.277860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.278197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.278421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] PATH_INFO: `/v3/projects/3bc7a35bc59041c3bed1c8bbea6f6bc2` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.279539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3bc7a35bc59041c3bed1c8bbea6f6bc2)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:15.281531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:15.318007 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:15.333020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3bc7a35bc59041c3bed1c8bbea6f6bc2', 'request_id': 'req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:15.340371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3bc7a35bc59041c3bed1c8bbea6f6bc2', 'request_id': 'req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:15.346080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3bc7a35bc59041c3bed1c8bbea6f6bc2', 'request_id': 'req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:15.357077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] The token cache is being invalidate because project 3bc7a35bc59041c3bed1c8bbea6f6bc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:19:15.357366 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:15.358003 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-00fc7ae7-a6b3-4395-87cb-3bb8bcac41f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:15.359499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1262/2511] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:19:15 2022] DELETE /identity/v3/projects/3bc7a35bc59041c3bed1c8bbea6f6bc2 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:15.365242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf264971-1762-4679-bcd1-2880634c9cce None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.365524 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf264971-1762-4679-bcd1-2880634c9cce None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.365798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf264971-1762-4679-bcd1-2880634c9cce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.385596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-bf264971-1762-4679-bcd1-2880634c9cce None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:15.386793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bf264971-1762-4679-bcd1-2880634c9cce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:15.446283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1250/2512] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:15.454335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23fd8df-4a5b-479c-b06a-c66f2a63e310 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.454618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23fd8df-4a5b-479c-b06a-c66f2a63e310 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.454899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23fd8df-4a5b-479c-b06a-c66f2a63e310 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.474991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d23fd8df-4a5b-479c-b06a-c66f2a63e310 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:15.476267 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d23fd8df-4a5b-479c-b06a-c66f2a63e310 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:15.485018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1263/2513] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:15.489154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf264971-1762-4679-bcd1-2880634c9cce None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:15.498753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf264971-1762-4679-bcd1-2880634c9cce None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:15.503366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:15.503997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.504279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.504550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.505190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:19:15.505520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:15.507098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:15.518562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1251/2514] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:19:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:19:15.522628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23fd8df-4a5b-479c-b06a-c66f2a63e310 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:15.532873 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23fd8df-4a5b-479c-b06a-c66f2a63e310 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:15.545868 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663b7099-d8a3-431a-b2cf-153b442c71c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:15.547145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663b7099-d8a3-431a-b2cf-153b442c71c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.547439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663b7099-d8a3-431a-b2cf-153b442c71c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.547727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663b7099-d8a3-431a-b2cf-153b442c71c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.548362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663b7099-d8a3-431a-b2cf-153b442c71c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:15.554143 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663b7099-d8a3-431a-b2cf-153b442c71c0 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:15.586137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1264/2515] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:19:15 2022] POST /identity/v3/projects => generated 359 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:19:15.590267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:15.600938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fd905-c734-4b87-8c8b-4d91f5f9c9a1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:15.655192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e61b6-e87e-4586-8958-4f5f7be36413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:15.656433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e61b6-e87e-4586-8958-4f5f7be36413 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.656711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e61b6-e87e-4586-8958-4f5f7be36413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.656985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e61b6-e87e-4586-8958-4f5f7be36413 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.657622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e61b6-e87e-4586-8958-4f5f7be36413 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:15.659534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e61b6-e87e-4586-8958-4f5f7be36413 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:15.680091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1252/2516] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:19:15 2022] POST /identity/v3/users => generated 280 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:19:15.684380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663b7099-d8a3-431a-b2cf-153b442c71c0 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:15.695557 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663b7099-d8a3-431a-b2cf-153b442c71c0 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:15.700106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6651c2b8-ec87-4e82-9f52-b16b4978a230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:15.700693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6651c2b8-ec87-4e82-9f52-b16b4978a230 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.700971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6651c2b8-ec87-4e82-9f52-b16b4978a230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.701242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6651c2b8-ec87-4e82-9f52-b16b4978a230 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.701878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6651c2b8-ec87-4e82-9f52-b16b4978a230 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:15.704202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6651c2b8-ec87-4e82-9f52-b16b4978a230 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:15.714047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1265/2517] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:19:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:19:15.718786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e61b6-e87e-4586-8958-4f5f7be36413 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:15.728656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e61b6-e87e-4586-8958-4f5f7be36413 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:15.733199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9024ff96-96f0-491e-8dc7-6da9be8fcdcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:15.734100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ff96-96f0-491e-8dc7-6da9be8fcdcd admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.734374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ff96-96f0-491e-8dc7-6da9be8fcdcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.734646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ff96-96f0-491e-8dc7-6da9be8fcdcd admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.735282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9024ff96-96f0-491e-8dc7-6da9be8fcdcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:15.736883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9024ff96-96f0-491e-8dc7-6da9be8fcdcd admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:15.747239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1253/2518] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:19:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:19:15.751424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6651c2b8-ec87-4e82-9f52-b16b4978a230 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:15.761089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6651c2b8-ec87-4e82-9f52-b16b4978a230 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:15.766285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:15.766919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.767196 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.767469 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] PATH_INFO: `/v3/projects/b1b889e2f77e42cbaf5aa18b215077b9/users/c30a26797df547458a4e93567b2536d2/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.782705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1b889e2f77e42cbaf5aa18b215077b9, user_id=c30a26797df547458a4e93567b2536d2, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:15.785450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:15.797870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1266/2519] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:19:15 2022] PUT /identity/v3/projects/b1b889e2f77e42cbaf5aa18b215077b9/users/c30a26797df547458a4e93567b2536d2/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:19:15.809134 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a563386-0c2c-402d-aa73-445ff5c064af None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:15.809419 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a563386-0c2c-402d-aa73-445ff5c064af None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:15.809692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a563386-0c2c-402d-aa73-445ff5c064af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:15.817843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:15.840072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-8a563386-0c2c-402d-aa73-445ff5c064af None None] MFA Rules not processed for user `c30a26797df547458a4e93567b2536d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:15.841283 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8a563386-0c2c-402d-aa73-445ff5c064af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:15.907593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1254/2520] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:15 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:15.941240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:15.942993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:16.002528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ad4db2-b185-460c-abdf-e72a8786a08d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:16.006212 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b615c2f4-3177-4582-9645-3012f63edf02 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:16.006805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615c2f4-3177-4582-9645-3012f63edf02 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.007084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615c2f4-3177-4582-9645-3012f63edf02 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.007356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b615c2f4-3177-4582-9645-3012f63edf02 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.016293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b615c2f4-3177-4582-9645-3012f63edf02 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:16.017707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b615c2f4-3177-4582-9645-3012f63edf02 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:16.026540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1267/2521] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:15 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:16.041771 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a563386-0c2c-402d-aa73-445ff5c064af None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:16.105279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a563386-0c2c-402d-aa73-445ff5c064af None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:16.109864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:16.110499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.110780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.111055 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] PATH_INFO: `/v3/users/c30a26797df547458a4e93567b2536d2` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.112227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=c30a26797df547458a4e93567b2536d2)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:16.114202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:16.162494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c30a26797df547458a4e93567b2536d2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.170567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c30a26797df547458a4e93567b2536d2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.171306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c30a26797df547458a4e93567b2536d2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.171944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c30a26797df547458a4e93567b2536d2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.179230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c30a26797df547458a4e93567b2536d2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.197684 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1255/2522] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:19:16 2022] DELETE /identity/v3/users/c30a26797df547458a4e93567b2536d2 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:16.202080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d673df64-b026-430d-bb45-1b09f6e4a622 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.202899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d673df64-b026-430d-bb45-1b09f6e4a622 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.203178 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d673df64-b026-430d-bb45-1b09f6e4a622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.223776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d673df64-b026-430d-bb45-1b09f6e4a622 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:16.224974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d673df64-b026-430d-bb45-1b09f6e4a622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:16.277315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:16.288204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1268/2523] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:16.298239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:16.299584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:16.314073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:16.358330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18cf650-a751-4334-80fd-afae3c7cfe0c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:16.362508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2c7146-8c46-4360-8690-ba19787cb49c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:16.363109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2c7146-8c46-4360-8690-ba19787cb49c service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.363388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2c7146-8c46-4360-8690-ba19787cb49c service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.363660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2c7146-8c46-4360-8690-ba19787cb49c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.372521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2c7146-8c46-4360-8690-ba19787cb49c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:16.374891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2c7146-8c46-4360-8690-ba19787cb49c service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:16.384239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1256/2524] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:16.656377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d673df64-b026-430d-bb45-1b09f6e4a622 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:16.657802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d673df64-b026-430d-bb45-1b09f6e4a622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:16.669784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d673df64-b026-430d-bb45-1b09f6e4a622 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:16.675099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:16.676541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.676993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.677281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] PATH_INFO: `/v3/projects/b1b889e2f77e42cbaf5aa18b215077b9` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.678418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1b889e2f77e42cbaf5aa18b215077b9)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:16.680946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:16.730635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1b889e2f77e42cbaf5aa18b215077b9', 'request_id': 'req-21faab10-52f7-4e70-beda-887a494cde7d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.738986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1b889e2f77e42cbaf5aa18b215077b9', 'request_id': 'req-21faab10-52f7-4e70-beda-887a494cde7d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.744757 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1b889e2f77e42cbaf5aa18b215077b9', 'request_id': 'req-21faab10-52f7-4e70-beda-887a494cde7d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.756703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] The token cache is being invalidate because project b1b889e2f77e42cbaf5aa18b215077b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:19:16.756996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.757627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-21faab10-52f7-4e70-beda-887a494cde7d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:16.759087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1269/2525] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:19:16 2022] DELETE /identity/v3/projects/b1b889e2f77e42cbaf5aa18b215077b9 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:16.763934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe156e-8ba8-4562-adb7-f8773c3695fe None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.764384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe156e-8ba8-4562-adb7-f8773c3695fe None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.764478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe156e-8ba8-4562-adb7-f8773c3695fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.785854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-1afe156e-8ba8-4562-adb7-f8773c3695fe None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:16.787034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1afe156e-8ba8-4562-adb7-f8773c3695fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:16.845891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1257/2526] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:16.853474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600ad28-c05a-4e6d-bb0f-9c9e2a40ab0e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.853840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600ad28-c05a-4e6d-bb0f-9c9e2a40ab0e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.854121 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d600ad28-c05a-4e6d-bb0f-9c9e2a40ab0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.874003 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d600ad28-c05a-4e6d-bb0f-9c9e2a40ab0e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:16.875193 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d600ad28-c05a-4e6d-bb0f-9c9e2a40ab0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:16.883040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1270/2527] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:16.887085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe156e-8ba8-4562-adb7-f8773c3695fe None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:16.897099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe156e-8ba8-4562-adb7-f8773c3695fe None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:16.901593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:16.902200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.902460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.902728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.903372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:19:16.903670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:16.905304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:16.917333 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1258/2528] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:19:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:19:16.921605 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d600ad28-c05a-4e6d-bb0f-9c9e2a40ab0e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:16.930744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d600ad28-c05a-4e6d-bb0f-9c9e2a40ab0e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:16.935359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dff57d-77ea-467c-aad6-a32f6c7c07c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:16.935977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dff57d-77ea-467c-aad6-a32f6c7c07c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:16.936257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dff57d-77ea-467c-aad6-a32f6c7c07c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:16.936530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dff57d-77ea-467c-aad6-a32f6c7c07c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:16.937838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73dff57d-77ea-467c-aad6-a32f6c7c07c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:16.939786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73dff57d-77ea-467c-aad6-a32f6c7c07c2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:16.969603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1271/2529] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:19:16 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:19:16.973870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:16.984239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac77b6-0fc0-4602-bbfd-b40e9d3c61d1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.040059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3c3cb-e856-42ca-8199-eafcc2e8913b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.040655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3c3cb-e856-42ca-8199-eafcc2e8913b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.040930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3c3cb-e856-42ca-8199-eafcc2e8913b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.041195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3c3cb-e856-42ca-8199-eafcc2e8913b admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.041825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3c3cb-e856-42ca-8199-eafcc2e8913b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.043748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3c3cb-e856-42ca-8199-eafcc2e8913b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.065572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1259/2530] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:19:16 2022] POST /identity/v3/users => generated 288 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:19:17.069686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dff57d-77ea-467c-aad6-a32f6c7c07c2 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.080376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dff57d-77ea-467c-aad6-a32f6c7c07c2 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.085044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cf42e6-cf21-48b2-8950-2e01953de120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.085639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf42e6-cf21-48b2-8950-2e01953de120 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.085919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf42e6-cf21-48b2-8950-2e01953de120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.086189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf42e6-cf21-48b2-8950-2e01953de120 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.086832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cf42e6-cf21-48b2-8950-2e01953de120 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.088475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cf42e6-cf21-48b2-8950-2e01953de120 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.098374 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1272/2531] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:19:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:19:17.102211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3c3cb-e856-42ca-8199-eafcc2e8913b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.112704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3c3cb-e856-42ca-8199-eafcc2e8913b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.117294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b58acd1-6a08-4696-953a-c432e5b3ffb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.117872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b58acd1-6a08-4696-953a-c432e5b3ffb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.118146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b58acd1-6a08-4696-953a-c432e5b3ffb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.118412 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b58acd1-6a08-4696-953a-c432e5b3ffb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.119039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b58acd1-6a08-4696-953a-c432e5b3ffb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.120639 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b58acd1-6a08-4696-953a-c432e5b3ffb2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.131936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1260/2532] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:19:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:19:17.135865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cf42e6-cf21-48b2-8950-2e01953de120 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.145763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cf42e6-cf21-48b2-8950-2e01953de120 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.150361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f2a8c5-0531-4ddf-bf37-4c5b4d9a903f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.150996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f2a8c5-0531-4ddf-bf37-4c5b4d9a903f admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.151274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f2a8c5-0531-4ddf-bf37-4c5b4d9a903f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.151550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f2a8c5-0531-4ddf-bf37-4c5b4d9a903f admin admin] PATH_INFO: `/v3/projects/a4a6d9e2d9cd4fffaba2fa8ca886e7c0/users/0cb98f64b6a84a25901229a905fd2585/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.165898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f2a8c5-0531-4ddf-bf37-4c5b4d9a903f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4a6d9e2d9cd4fffaba2fa8ca886e7c0, user_id=0cb98f64b6a84a25901229a905fd2585, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.168600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f2a8c5-0531-4ddf-bf37-4c5b4d9a903f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.182396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1273/2533] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:19:17 2022] PUT /identity/v3/projects/a4a6d9e2d9cd4fffaba2fa8ca886e7c0/users/0cb98f64b6a84a25901229a905fd2585/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:17.193456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c03b31-b52c-4b6a-b985-3f23006b70b3 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.193733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c03b31-b52c-4b6a-b985-3f23006b70b3 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.194005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c03b31-b52c-4b6a-b985-3f23006b70b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.222344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-42c03b31-b52c-4b6a-b985-3f23006b70b3 None None] MFA Rules not processed for user `0cb98f64b6a84a25901229a905fd2585`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:17.223529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-42c03b31-b52c-4b6a-b985-3f23006b70b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:17.232483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:17.289854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1261/2534] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:17.294319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f2a8c5-0531-4ddf-bf37-4c5b4d9a903f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.362976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f2a8c5-0531-4ddf-bf37-4c5b4d9a903f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.368100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d99cd0-c5df-422e-82df-ba553b2ae939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.368697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d99cd0-c5df-422e-82df-ba553b2ae939 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.368976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d99cd0-c5df-422e-82df-ba553b2ae939 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.369246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d99cd0-c5df-422e-82df-ba553b2ae939 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.369905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d99cd0-c5df-422e-82df-ba553b2ae939 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.371826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d99cd0-c5df-422e-82df-ba553b2ae939 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.402690 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1274/2535] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:19:17 2022] POST /identity/v3/projects => generated 375 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:19:17.403839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c03b31-b52c-4b6a-b985-3f23006b70b3 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.415501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c03b31-b52c-4b6a-b985-3f23006b70b3 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.472756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9409c80-9dc5-4208-8955-fd2ff645db7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.473367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9409c80-9dc5-4208-8955-fd2ff645db7e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.473642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9409c80-9dc5-4208-8955-fd2ff645db7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.473908 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9409c80-9dc5-4208-8955-fd2ff645db7e admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.474530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9409c80-9dc5-4208-8955-fd2ff645db7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.476484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9409c80-9dc5-4208-8955-fd2ff645db7e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.497598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1262/2536] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:19:17 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:19:17.501779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d99cd0-c5df-422e-82df-ba553b2ae939 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.508484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:17.512360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d99cd0-c5df-422e-82df-ba553b2ae939 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.518314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928127c-1ee5-4f4f-816b-fd8a9aab3b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.519169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928127c-1ee5-4f4f-816b-fd8a9aab3b0c admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.519618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928127c-1ee5-4f4f-816b-fd8a9aab3b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.519976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928127c-1ee5-4f4f-816b-fd8a9aab3b0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.520826 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b928127c-1ee5-4f4f-816b-fd8a9aab3b0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.522522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b928127c-1ee5-4f4f-816b-fd8a9aab3b0c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.533680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1275/2537] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:19:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:19:17.537701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9409c80-9dc5-4208-8955-fd2ff645db7e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.548436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9409c80-9dc5-4208-8955-fd2ff645db7e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.552862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcdc12c-406f-4ad6-8392-4326e73cebad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.553499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcdc12c-406f-4ad6-8392-4326e73cebad admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.553772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcdc12c-406f-4ad6-8392-4326e73cebad admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.554040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcdc12c-406f-4ad6-8392-4326e73cebad admin admin] PATH_INFO: `/v3/projects/ca2937539df545949f01d0907b5e3fb7/users/c3919ddd53df41b5974af65a9544d680/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.568816 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcdc12c-406f-4ad6-8392-4326e73cebad admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca2937539df545949f01d0907b5e3fb7, user_id=c3919ddd53df41b5974af65a9544d680, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.571732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcdc12c-406f-4ad6-8392-4326e73cebad admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.584225 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1263/2538] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:19:17 2022] PUT /identity/v3/projects/ca2937539df545949f01d0907b5e3fb7/users/c3919ddd53df41b5974af65a9544d680/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:17.588083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928127c-1ee5-4f4f-816b-fd8a9aab3b0c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.598634 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928127c-1ee5-4f4f-816b-fd8a9aab3b0c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.653280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28832dce-fbd4-4a9d-acc5-06943e7c768d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.653790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28832dce-fbd4-4a9d-acc5-06943e7c768d admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.654066 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28832dce-fbd4-4a9d-acc5-06943e7c768d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.654337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28832dce-fbd4-4a9d-acc5-06943e7c768d admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.654961 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28832dce-fbd4-4a9d-acc5-06943e7c768d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.657242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28832dce-fbd4-4a9d-acc5-06943e7c768d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.666677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1276/2539] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:19:17 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:19:17.670605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcdc12c-406f-4ad6-8392-4326e73cebad admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.681541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcdc12c-406f-4ad6-8392-4326e73cebad admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.686076 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.686811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.687262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.687651 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] PATH_INFO: `/v3/projects/ca2937539df545949f01d0907b5e3fb7/users/c3919ddd53df41b5974af65a9544d680/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.689569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca2937539df545949f01d0907b5e3fb7, user_id=c3919ddd53df41b5974af65a9544d680, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:17.692218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:17.702538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1264/2540] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:19:17 2022] PUT /identity/v3/projects/ca2937539df545949f01d0907b5e3fb7/users/c3919ddd53df41b5974af65a9544d680/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:17.766684 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a8324-dc45-4788-86eb-981d2726eb41 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.767258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a8324-dc45-4788-86eb-981d2726eb41 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.767258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a8324-dc45-4788-86eb-981d2726eb41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.795403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-083a8324-dc45-4788-86eb-981d2726eb41 None None] MFA Rules not processed for user `c3919ddd53df41b5974af65a9544d680`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:17.797395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-083a8324-dc45-4788-86eb-981d2726eb41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:17.866088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1277/2541] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:17 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:19:17.876851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:17.878660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:17.935810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c215c3-7064-472a-a60c-48a78c19645f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:17.940078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4183c3-75ee-4441-8e1a-542ec58189e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:17.940670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4183c3-75ee-4441-8e1a-542ec58189e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:17.940947 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4183c3-75ee-4441-8e1a-542ec58189e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:17.941216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4183c3-75ee-4441-8e1a-542ec58189e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:17.958188 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:17.998331 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4183c3-75ee-4441-8e1a-542ec58189e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:18.000473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4183c3-75ee-4441-8e1a-542ec58189e4 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:18.009090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1265/2542] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:17 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:20.541576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a8324-dc45-4788-86eb-981d2726eb41 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:20.602380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a8324-dc45-4788-86eb-981d2726eb41 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:20.606808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:20.607426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:20.607732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:20.608012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] PATH_INFO: `/v3/users/0cb98f64b6a84a25901229a905fd2585` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:20.609857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cb98f64b6a84a25901229a905fd2585)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:20.611809 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:20.627879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:20.656742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cb98f64b6a84a25901229a905fd2585'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:20.663565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cb98f64b6a84a25901229a905fd2585'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:20.664205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cb98f64b6a84a25901229a905fd2585'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:20.664805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cb98f64b6a84a25901229a905fd2585'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:20.672304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cb98f64b6a84a25901229a905fd2585'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:20.689651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1278/2543] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:19:20 2022] DELETE /identity/v3/users/0cb98f64b6a84a25901229a905fd2585 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:20.694088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07f679a-cd1d-4941-987f-9323b2effec1 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:20.694587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07f679a-cd1d-4941-987f-9323b2effec1 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:20.694750 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07f679a-cd1d-4941-987f-9323b2effec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:20.713798 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-a07f679a-cd1d-4941-987f-9323b2effec1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:20.714968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a07f679a-cd1d-4941-987f-9323b2effec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:20.773605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1266/2544] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:20.784298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:20.785710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:20.847332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6441795-cae1-4e8b-a543-4c2209ceb52c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:20.851030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5def83-375d-4366-9bf9-eb08ec504f05 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:20.851638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5def83-375d-4366-9bf9-eb08ec504f05 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:20.851950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5def83-375d-4366-9bf9-eb08ec504f05 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:20.852228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5def83-375d-4366-9bf9-eb08ec504f05 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:20.861658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5def83-375d-4366-9bf9-eb08ec504f05 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:20.863026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5def83-375d-4366-9bf9-eb08ec504f05 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:20.872902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1279/2545] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:21.142658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07f679a-cd1d-4941-987f-9323b2effec1 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:21.144159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-a07f679a-cd1d-4941-987f-9323b2effec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:21.157537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07f679a-cd1d-4941-987f-9323b2effec1 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:21.162204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:21.162865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:21.163143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:21.163415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] PATH_INFO: `/v3/projects/a4a6d9e2d9cd4fffaba2fa8ca886e7c0` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:21.164562 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4a6d9e2d9cd4fffaba2fa8ca886e7c0)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:21.166518 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:21.212858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4a6d9e2d9cd4fffaba2fa8ca886e7c0', 'request_id': 'req-e73bd0dc-0b84-4631-bc9f-039dd7033b20'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.220843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4a6d9e2d9cd4fffaba2fa8ca886e7c0', 'request_id': 'req-e73bd0dc-0b84-4631-bc9f-039dd7033b20'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.226407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4a6d9e2d9cd4fffaba2fa8ca886e7c0', 'request_id': 'req-e73bd0dc-0b84-4631-bc9f-039dd7033b20'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.237937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] The token cache is being invalidate because project a4a6d9e2d9cd4fffaba2fa8ca886e7c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:19:21.238369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.238903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.240430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1267/2546] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:19:21 2022] DELETE /identity/v3/projects/a4a6d9e2d9cd4fffaba2fa8ca886e7c0 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:21.244394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5def83-375d-4366-9bf9-eb08ec504f05 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:21.254040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5def83-375d-4366-9bf9-eb08ec504f05 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:21.313543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:21.314173 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:21.314456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:21.314726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] PATH_INFO: `/v3/users/c3919ddd53df41b5974af65a9544d680` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:21.315888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3919ddd53df41b5974af65a9544d680)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:21.317852 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:21.359297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:21.365625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3919ddd53df41b5974af65a9544d680'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.373180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3919ddd53df41b5974af65a9544d680'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.373831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3919ddd53df41b5974af65a9544d680'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.375037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3919ddd53df41b5974af65a9544d680'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.382033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-bc207800-118a-44db-9cc4-38a8c0d26402 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3919ddd53df41b5974af65a9544d680'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.399891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1280/2547] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:19:21 2022] DELETE /identity/v3/users/c3919ddd53df41b5974af65a9544d680 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:21.672144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:21.673537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:21.688728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73bd0dc-0b84-4631-bc9f-039dd7033b20 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:21.730427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:21.748627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:21.749332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:21.749611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:21.750023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] PATH_INFO: `/v3/projects/ca2937539df545949f01d0907b5e3fb7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:21.751258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca2937539df545949f01d0907b5e3fb7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:21.753192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:21.798843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca2937539df545949f01d0907b5e3fb7', 'request_id': 'req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.806292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca2937539df545949f01d0907b5e3fb7', 'request_id': 'req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.811936 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca2937539df545949f01d0907b5e3fb7', 'request_id': 'req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.823397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] The token cache is being invalidate because project ca2937539df545949f01d0907b5e3fb7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:19:21.823808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.824445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-1e860438-f5ad-44df-8af7-3d6ca2e1ae22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:21.826149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1268/2548] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:19:21 2022] DELETE /identity/v3/projects/ca2937539df545949f01d0907b5e3fb7 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:19:21.831386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546a2f11-2560-4204-b817-259302b433da None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:21.831874 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546a2f11-2560-4204-b817-259302b433da None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:21.831995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546a2f11-2560-4204-b817-259302b433da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:21.852182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-546a2f11-2560-4204-b817-259302b433da None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:21.853376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-546a2f11-2560-4204-b817-259302b433da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:21.912109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1281/2549] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:21.919899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8837efe-c49e-4fa8-ada2-418921d72935 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:21.920120 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8837efe-c49e-4fa8-ada2-418921d72935 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:21.920392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8837efe-c49e-4fa8-ada2-418921d72935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:21.940130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c8837efe-c49e-4fa8-ada2-418921d72935 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:21.941308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c8837efe-c49e-4fa8-ada2-418921d72935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:21.949317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1269/2550] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:21.953786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546a2f11-2560-4204-b817-259302b433da None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:21.963527 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546a2f11-2560-4204-b817-259302b433da None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:21.968762 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:21.969356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:21.969638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:21.969910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:21.970550 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:19:21.970854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:21.972485 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:21.984976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1282/2551] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:19:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 14:19:21.989324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8837efe-c49e-4fa8-ada2-418921d72935 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:21.998589 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8837efe-c49e-4fa8-ada2-418921d72935 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:22.003573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dd6afa-668d-4028-90f8-b3e4d5f5b003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:22.004268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dd6afa-668d-4028-90f8-b3e4d5f5b003 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:22.004626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dd6afa-668d-4028-90f8-b3e4d5f5b003 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:22.004962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dd6afa-668d-4028-90f8-b3e4d5f5b003 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:22.005671 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dd6afa-668d-4028-90f8-b3e4d5f5b003 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:22.007619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dd6afa-668d-4028-90f8-b3e4d5f5b003 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:22.035382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1270/2552] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:19:21 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:19:22.039666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:22.050627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9606b324-3b8f-4c49-b846-7243b1c2c4bd admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:22.108396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059eaa03-8687-4e61-aeca-1c000cea0c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:22.110059 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059eaa03-8687-4e61-aeca-1c000cea0c92 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:22.110420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059eaa03-8687-4e61-aeca-1c000cea0c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:22.110697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059eaa03-8687-4e61-aeca-1c000cea0c92 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:22.111330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059eaa03-8687-4e61-aeca-1c000cea0c92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:22.113344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059eaa03-8687-4e61-aeca-1c000cea0c92 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:22.135028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1283/2553] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:19:22 2022] POST /identity/v3/users => generated 288 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:19:22.138938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dd6afa-668d-4028-90f8-b3e4d5f5b003 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:22.149996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dd6afa-668d-4028-90f8-b3e4d5f5b003 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:22.154910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb1050-30e5-4747-93d2-93e0fc926bcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:22.156079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb1050-30e5-4747-93d2-93e0fc926bcf admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:22.156365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb1050-30e5-4747-93d2-93e0fc926bcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:22.156802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb1050-30e5-4747-93d2-93e0fc926bcf admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:22.157459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadb1050-30e5-4747-93d2-93e0fc926bcf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:22.159048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadb1050-30e5-4747-93d2-93e0fc926bcf admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:22.169000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1271/2554] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:19:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:19:22.173314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059eaa03-8687-4e61-aeca-1c000cea0c92 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:22.183148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059eaa03-8687-4e61-aeca-1c000cea0c92 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:22.188360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba05eb-44de-4c66-9c99-33b9eb464d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:22.188949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba05eb-44de-4c66-9c99-33b9eb464d5c admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:22.189333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba05eb-44de-4c66-9c99-33b9eb464d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:22.189506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba05eb-44de-4c66-9c99-33b9eb464d5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:22.190137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba05eb-44de-4c66-9c99-33b9eb464d5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:22.191766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba05eb-44de-4c66-9c99-33b9eb464d5c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:22.203060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1284/2555] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:19:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:19:22.207149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb1050-30e5-4747-93d2-93e0fc926bcf admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:22.217092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb1050-30e5-4747-93d2-93e0fc926bcf admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:22.221905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:22.222546 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:22.222822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:22.223092 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] PATH_INFO: `/v3/projects/45248c0742754b3b8228dd9fa2616724/users/109e32e9a6a14c22811dd6cacb6e2e56/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:22.238315 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] RBAC: Authorizing `identity:create_grant(project_id=45248c0742754b3b8228dd9fa2616724, user_id=109e32e9a6a14c22811dd6cacb6e2e56, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:22.240978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:22.254373 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1272/2556] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:19:22 2022] PUT /identity/v3/projects/45248c0742754b3b8228dd9fa2616724/users/109e32e9a6a14c22811dd6cacb6e2e56/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:22.266281 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c79c52-d613-4ff8-9a81-7fecba6a4bf1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:22.266564 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c79c52-d613-4ff8-9a81-7fecba6a4bf1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:22.266834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c79c52-d613-4ff8-9a81-7fecba6a4bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:22.294220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a4c79c52-d613-4ff8-9a81-7fecba6a4bf1 None None] MFA Rules not processed for user `109e32e9a6a14c22811dd6cacb6e2e56`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:22.295405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a4c79c52-d613-4ff8-9a81-7fecba6a4bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:22.305489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:22.362146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1285/2557] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:22.372680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:22.374995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:22.433476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbaf1ec-3b12-46a8-bcde-2476cbf12454 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:22.437124 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5505cc-afcc-4c6a-9e4a-f153340d1518 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:22.437731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5505cc-afcc-4c6a-9e4a-f153340d1518 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:22.438010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5505cc-afcc-4c6a-9e4a-f153340d1518 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:22.438274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5505cc-afcc-4c6a-9e4a-f153340d1518 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:22.446818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5505cc-afcc-4c6a-9e4a-f153340d1518 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:22.448330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5505cc-afcc-4c6a-9e4a-f153340d1518 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:22.458766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1273/2558] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:22 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:23.821932 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd3d9e4-48a7-41c4-bbdf-a30b0e9addd9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:23.822436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd3d9e4-48a7-41c4-bbdf-a30b0e9addd9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:23.822538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd3d9e4-48a7-41c4-bbdf-a30b0e9addd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:23.842053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2dd3d9e4-48a7-41c4-bbdf-a30b0e9addd9 None None] MFA Rules not processed for user `109e32e9a6a14c22811dd6cacb6e2e56`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:23.843819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2dd3d9e4-48a7-41c4-bbdf-a30b0e9addd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:23.850526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1286/2559] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:32.038014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1f2a6-e5b4-4a27-bdbb-0bd8888ba0da None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:32.038506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1f2a6-e5b4-4a27-bdbb-0bd8888ba0da None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:32.038625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae1f2a6-e5b4-4a27-bdbb-0bd8888ba0da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:32.058853 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-3ae1f2a6-e5b4-4a27-bdbb-0bd8888ba0da None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:32.060160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3ae1f2a6-e5b4-4a27-bdbb-0bd8888ba0da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:32.065762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:32.118421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1274/2560] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:19:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:32.166993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd3d9e4-48a7-41c4-bbdf-a30b0e9addd9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:32.168472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2dd3d9e4-48a7-41c4-bbdf-a30b0e9addd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:32.241391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd3d9e4-48a7-41c4-bbdf-a30b0e9addd9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:32.246419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:32.247034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:32.247315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:32.247588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:32.256821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:32.258434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:32.267332 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1287/2561] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:19:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:19:37.818509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae1f2a6-e5b4-4a27-bdbb-0bd8888ba0da None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:37.820157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3ae1f2a6-e5b4-4a27-bdbb-0bd8888ba0da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:37.883854 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae1f2a6-e5b4-4a27-bdbb-0bd8888ba0da None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:37.887453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93defcb9-f311-4939-bc0a-33e5463fd5e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:37.888098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93defcb9-f311-4939-bc0a-33e5463fd5e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:37.888383 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93defcb9-f311-4939-bc0a-33e5463fd5e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:37.888658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93defcb9-f311-4939-bc0a-33e5463fd5e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:37.890431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-93defcb9-f311-4939-bc0a-33e5463fd5e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:37.993752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93defcb9-f311-4939-bc0a-33e5463fd5e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:37.995445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93defcb9-f311-4939-bc0a-33e5463fd5e5 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:38.023552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1275/2562] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:38.092041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:38.095096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:38.122898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174a8b0-887f-4995-88f0-683414f8325b service swift] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:38.124067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:38.124986 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:38.125268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:38.125619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:38.128116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:38.146132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:38.147582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:38.163052 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1288/2563] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:40.694962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de05b11-b3c3-42fc-ac11-8279e4549c32 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:40.695505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de05b11-b3c3-42fc-ac11-8279e4549c32 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:40.695662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de05b11-b3c3-42fc-ac11-8279e4549c32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:40.717501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6de05b11-b3c3-42fc-ac11-8279e4549c32 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:40.718776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6de05b11-b3c3-42fc-ac11-8279e4549c32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:40.726699 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1276/2564] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:19:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:40.740284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:40.752650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cea1de2-ac0b-4448-af09-d2ced88842c2 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:40.757705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337febd2-ce41-466d-8dc0-97fe8ca229f4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:40.758327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337febd2-ce41-466d-8dc0-97fe8ca229f4 service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:40.758617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337febd2-ce41-466d-8dc0-97fe8ca229f4 service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:40.758891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337febd2-ce41-466d-8dc0-97fe8ca229f4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:40.768609 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337febd2-ce41-466d-8dc0-97fe8ca229f4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:40.770231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337febd2-ce41-466d-8dc0-97fe8ca229f4 service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:40.779987 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1289/2565] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:19:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:19:41.586878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4829d4c-4c05-4be1-b9dd-e6a7cd1da228 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:41.587167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4829d4c-4c05-4be1-b9dd-e6a7cd1da228 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:41.587445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4829d4c-4c05-4be1-b9dd-e6a7cd1da228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:41.607326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b4829d4c-4c05-4be1-b9dd-e6a7cd1da228 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:41.608677 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b4829d4c-4c05-4be1-b9dd-e6a7cd1da228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:41.621306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1277/2566] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 15 14:19:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:41.632106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337febd2-ce41-466d-8dc0-97fe8ca229f4 service swift] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:41.644974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337febd2-ce41-466d-8dc0-97fe8ca229f4 service swift] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:41.650930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1aec2b-ab44-40f0-945c-7db99487f287 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:41.651603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1aec2b-ab44-40f0-945c-7db99487f287 service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:41.652017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1aec2b-ab44-40f0-945c-7db99487f287 service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:41.652352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1aec2b-ab44-40f0-945c-7db99487f287 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:41.661265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1aec2b-ab44-40f0-945c-7db99487f287 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:41.663114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1aec2b-ab44-40f0-945c-7db99487f287 service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:41.672492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1290/2567] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:19:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:19:41.962223 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d718dbc-1a7b-4a0f-b04d-22b1b262e27d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:41.962726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d718dbc-1a7b-4a0f-b04d-22b1b262e27d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:41.962829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d718dbc-1a7b-4a0f-b04d-22b1b262e27d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:41.981857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6d718dbc-1a7b-4a0f-b04d-22b1b262e27d None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:41.983048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6d718dbc-1a7b-4a0f-b04d-22b1b262e27d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:41.990866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1278/2568] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:19:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:42.003537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1aec2b-ab44-40f0-945c-7db99487f287 service swift] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:42.014019 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1aec2b-ab44-40f0-945c-7db99487f287 service swift] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:42.019087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6a60e-beaa-4738-b1e9-c187743cc42c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:42.019672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6a60e-beaa-4738-b1e9-c187743cc42c service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:42.020014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6a60e-beaa-4738-b1e9-c187743cc42c service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:42.020288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6a60e-beaa-4738-b1e9-c187743cc42c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:42.028164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f6a60e-beaa-4738-b1e9-c187743cc42c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:42.029734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f6a60e-beaa-4738-b1e9-c187743cc42c service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:42.037908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1291/2569] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:19:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:19:42.351372 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e90829f-3434-412c-8dda-3c19c7087a23 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:42.351654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e90829f-3434-412c-8dda-3c19c7087a23 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:42.351960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e90829f-3434-412c-8dda-3c19c7087a23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:42.370768 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6e90829f-3434-412c-8dda-3c19c7087a23 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:42.371997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6e90829f-3434-412c-8dda-3c19c7087a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:42.379670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1279/2570] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 15 14:19:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:42.392226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6a60e-beaa-4738-b1e9-c187743cc42c service swift] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:42.402546 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6a60e-beaa-4738-b1e9-c187743cc42c service swift] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:42.407506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64d03ee-2308-4bcc-b123-f79d9e17b717 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:42.408127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64d03ee-2308-4bcc-b123-f79d9e17b717 service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:42.408408 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64d03ee-2308-4bcc-b123-f79d9e17b717 service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:42.408679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64d03ee-2308-4bcc-b123-f79d9e17b717 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:42.417067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64d03ee-2308-4bcc-b123-f79d9e17b717 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:42.418730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64d03ee-2308-4bcc-b123-f79d9e17b717 service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:42.427419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1292/2571] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:19:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:19:44.550814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e90829f-3434-412c-8dda-3c19c7087a23 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:44.562761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:44.612983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e90829f-3434-412c-8dda-3c19c7087a23 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:44.617377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:44.617994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:44.618272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:44.618541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] PATH_INFO: `/v3/users/109e32e9a6a14c22811dd6cacb6e2e56` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:44.619650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=109e32e9a6a14c22811dd6cacb6e2e56)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:44.621574 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:44.666512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '109e32e9a6a14c22811dd6cacb6e2e56'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:44.673689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '109e32e9a6a14c22811dd6cacb6e2e56'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:44.674320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '109e32e9a6a14c22811dd6cacb6e2e56'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:44.674931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '109e32e9a6a14c22811dd6cacb6e2e56'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:44.681875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '109e32e9a6a14c22811dd6cacb6e2e56'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:44.699242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1280/2572] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:19:44 2022] DELETE /identity/v3/users/109e32e9a6a14c22811dd6cacb6e2e56 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:44.704474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1453a-03a2-470d-9981-5a96b7754418 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:44.704959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1453a-03a2-470d-9981-5a96b7754418 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:44.705074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d1453a-03a2-470d-9981-5a96b7754418 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:44.725255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c5d1453a-03a2-470d-9981-5a96b7754418 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:44.726538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c5d1453a-03a2-470d-9981-5a96b7754418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:44.750445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:44.786478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1293/2573] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:19:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:44.796194 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:44.797776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:44.855107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe5d10-2c74-47eb-9d84-1247c177fe4e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:44.858755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5912cf-d535-49df-9c85-19070449bf3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:44.859355 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5912cf-d535-49df-9c85-19070449bf3e service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:44.859663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5912cf-d535-49df-9c85-19070449bf3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:44.860024 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5912cf-d535-49df-9c85-19070449bf3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:44.869038 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5912cf-d535-49df-9c85-19070449bf3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:44.870463 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5912cf-d535-49df-9c85-19070449bf3e service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:44.880056 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1281/2574] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:45.049139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d1453a-03a2-470d-9981-5a96b7754418 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:45.050620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c5d1453a-03a2-470d-9981-5a96b7754418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:45.069845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d1453a-03a2-470d-9981-5a96b7754418 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:45.075729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:45.076362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:45.076713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:45.076906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] PATH_INFO: `/v3/projects/45248c0742754b3b8228dd9fa2616724` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:45.078198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] RBAC: Authorizing `identity:delete_project(project_id=45248c0742754b3b8228dd9fa2616724)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:45.080812 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:45.128741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45248c0742754b3b8228dd9fa2616724', 'request_id': 'req-7db8adee-ca17-4b70-85f1-39d8a8202985'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:45.136030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45248c0742754b3b8228dd9fa2616724', 'request_id': 'req-7db8adee-ca17-4b70-85f1-39d8a8202985'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:45.142514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45248c0742754b3b8228dd9fa2616724', 'request_id': 'req-7db8adee-ca17-4b70-85f1-39d8a8202985'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:45.154164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] The token cache is being invalidate because project 45248c0742754b3b8228dd9fa2616724 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:19:45.154453 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:45.155099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:19:45.156948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1294/2575] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:19:45 2022] DELETE /identity/v3/projects/45248c0742754b3b8228dd9fa2616724 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:19:48.637529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c475e8b3-2201-4325-a172-939c97cd1650 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:48.638008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c475e8b3-2201-4325-a172-939c97cd1650 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:48.638105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c475e8b3-2201-4325-a172-939c97cd1650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:48.657423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c475e8b3-2201-4325-a172-939c97cd1650 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:19:48.659289 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c475e8b3-2201-4325-a172-939c97cd1650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:48.688645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:19:48.717102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1282/2576] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 15 14:19:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:19:48.729927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:48.731492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:48.803575 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db8adee-ca17-4b70-85f1-39d8a8202985 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:48.808872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c8fea9-3482-4ccc-aca1-bd7811a1b32c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:48.809602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c8fea9-3482-4ccc-aca1-bd7811a1b32c service swift] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:48.809893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c8fea9-3482-4ccc-aca1-bd7811a1b32c service swift] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:48.810172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c8fea9-3482-4ccc-aca1-bd7811a1b32c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:48.819141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c8fea9-3482-4ccc-aca1-bd7811a1b32c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:48.820912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c8fea9-3482-4ccc-aca1-bd7811a1b32c service swift] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:48.829427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1295/2577] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:19:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:19:57.102374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c475e8b3-2201-4325-a172-939c97cd1650 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:19:57.103782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c475e8b3-2201-4325-a172-939c97cd1650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:57.183376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c475e8b3-2201-4325-a172-939c97cd1650 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:19:57.188822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:19:57.189451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:19:57.189734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:19:57.190009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:19:57.191769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:19:57.272735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:19:57.274453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:19:57.292584 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1283/2578] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:19:57 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:19:58.826104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=73,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:07.289949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=79 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:07.858872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cc183-da71-4790-8b56-377bd0be5524 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:07.859355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cc183-da71-4790-8b56-377bd0be5524 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:07.859454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cc183-da71-4790-8b56-377bd0be5524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:07.907892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-112cc183-da71-4790-8b56-377bd0be5524 None None] MFA Rules not processed for user `75c8381b6ad84cac881db621d2d99af7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:07.909089 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-112cc183-da71-4790-8b56-377bd0be5524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:07.925378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1296/2579] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 15 14:20:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:07.938393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:07.959856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87128909-26c3-4acc-ba47-ba1ee3ded11c service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:07.964801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc2a26b4fc4d24b984fa8097752055', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:07.965399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:07.965679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:07.965952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:07.973794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:07.975542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:07.983410 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1284/2580] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 15 14:20:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 15 14:20:17.922541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=18 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:17.980234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=10 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:32.371742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cc183-da71-4790-8b56-377bd0be5524 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:32.373139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-112cc183-da71-4790-8b56-377bd0be5524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:32.429225 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cc183-da71-4790-8b56-377bd0be5524 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:32.432827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:32.433429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:32.433709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:32.433980 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:32.435732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:32.498937 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:32.500420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:32.510810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1297/2581] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:20:35.578220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:35.579796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:35.594948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846d41ca-6b02-4dd4-8676-d0b8cd3bbdeb service swift] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:35.599449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:35.600116 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:35.600401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:35.601320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] PATH_INFO: `/v3/users/2049d66708ce47a794c14ad4c2040937` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:35.616015 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2049d66708ce47a794c14ad4c2040937)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:35.618196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:35.668100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2049d66708ce47a794c14ad4c2040937'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:35.675091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2049d66708ce47a794c14ad4c2040937'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:35.675814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2049d66708ce47a794c14ad4c2040937'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:35.676443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2049d66708ce47a794c14ad4c2040937'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:35.683632 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7929f88b-5931-4d04-b794-8206d5780c65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2049d66708ce47a794c14ad4c2040937'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:35.700652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1285/2582] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:20:35 2022] DELETE /identity/v3/users/2049d66708ce47a794c14ad4c2040937 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:35.872928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:35.885046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468fb9a5-45b9-4f53-9253-806c37baaaba service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:35.941875 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:35.942497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:35.942774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:35.943043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] PATH_INFO: `/v3/projects/a8297077982a4338a2d687c89e1119a2` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:35.954386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8297077982a4338a2d687c89e1119a2)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:35.956312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:35.998064 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:36.002749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8297077982a4338a2d687c89e1119a2', 'request_id': 'req-023b1aa6-db2b-42fb-982a-d9f522711824'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:36.010150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8297077982a4338a2d687c89e1119a2', 'request_id': 'req-023b1aa6-db2b-42fb-982a-d9f522711824'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:36.015854 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8297077982a4338a2d687c89e1119a2', 'request_id': 'req-023b1aa6-db2b-42fb-982a-d9f522711824'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:36.027871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] The token cache is being invalidate because project a8297077982a4338a2d687c89e1119a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:20:36.028163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:36.028795 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-023b1aa6-db2b-42fb-982a-d9f522711824 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:36.030256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1298/2583] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:20:35 2022] DELETE /identity/v3/projects/a8297077982a4338a2d687c89e1119a2 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:36.035279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe4c6da-f7fd-44e8-a419-5d2bac30e640 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.035568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe4c6da-f7fd-44e8-a419-5d2bac30e640 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.035879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe4c6da-f7fd-44e8-a419-5d2bac30e640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.056335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-1fe4c6da-f7fd-44e8-a419-5d2bac30e640 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:36.057459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1fe4c6da-f7fd-44e8-a419-5d2bac30e640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:36.115298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1286/2584] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:36.124793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238b5b1-b88f-4ebf-a4b9-b2ebb557fea7 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.125078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238b5b1-b88f-4ebf-a4b9-b2ebb557fea7 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.125348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238b5b1-b88f-4ebf-a4b9-b2ebb557fea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.144844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-1238b5b1-b88f-4ebf-a4b9-b2ebb557fea7 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:36.146048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1238b5b1-b88f-4ebf-a4b9-b2ebb557fea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:36.153923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1299/2585] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:36.158008 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe4c6da-f7fd-44e8-a419-5d2bac30e640 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:36.168342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe4c6da-f7fd-44e8-a419-5d2bac30e640 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:36.172849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:36.173443 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.173723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.173997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.174640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:20:36.174944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:36.176561 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:36.187973 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1287/2586] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:20:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:20:36.192152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1238b5b1-b88f-4ebf-a4b9-b2ebb557fea7 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:36.202507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1238b5b1-b88f-4ebf-a4b9-b2ebb557fea7 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:36.207200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f226d1-3229-40ff-8bdf-b761952df51b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:36.207830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f226d1-3229-40ff-8bdf-b761952df51b admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.208113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f226d1-3229-40ff-8bdf-b761952df51b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.208386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f226d1-3229-40ff-8bdf-b761952df51b admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.209024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f226d1-3229-40ff-8bdf-b761952df51b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:36.211655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f226d1-3229-40ff-8bdf-b761952df51b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:36.247857 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1300/2587] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:20:36 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:36.252152 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:36.262512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf11758c-300f-48bc-a568-b67045ab784f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:36.307523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:36.322467 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c9f7d-3393-4b16-9744-4f6eb53eec82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:36.323079 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c9f7d-3393-4b16-9744-4f6eb53eec82 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.323359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c9f7d-3393-4b16-9744-4f6eb53eec82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.323639 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c9f7d-3393-4b16-9744-4f6eb53eec82 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.324302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70c9f7d-3393-4b16-9744-4f6eb53eec82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:36.326200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70c9f7d-3393-4b16-9744-4f6eb53eec82 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:36.347033 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1288/2588] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:20:36 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:36.351355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f226d1-3229-40ff-8bdf-b761952df51b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:36.361512 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f226d1-3229-40ff-8bdf-b761952df51b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:36.366046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7ad1a7-a540-417b-aba5-dbcf54df1f04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:36.367230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7ad1a7-a540-417b-aba5-dbcf54df1f04 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.367498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7ad1a7-a540-417b-aba5-dbcf54df1f04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.367796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7ad1a7-a540-417b-aba5-dbcf54df1f04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.368434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7ad1a7-a540-417b-aba5-dbcf54df1f04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:36.370023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7ad1a7-a540-417b-aba5-dbcf54df1f04 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:36.380445 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1301/2589] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:36.384736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c9f7d-3393-4b16-9744-4f6eb53eec82 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:36.394975 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c9f7d-3393-4b16-9744-4f6eb53eec82 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:36.399713 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b75fe11-69ba-490f-a549-5134ad1c00aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:36.400328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b75fe11-69ba-490f-a549-5134ad1c00aa admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.400609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b75fe11-69ba-490f-a549-5134ad1c00aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.400892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b75fe11-69ba-490f-a549-5134ad1c00aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.401531 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b75fe11-69ba-490f-a549-5134ad1c00aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:36.403144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b75fe11-69ba-490f-a549-5134ad1c00aa admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:36.414048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1289/2590] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:36.418237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7ad1a7-a540-417b-aba5-dbcf54df1f04 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:36.428205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7ad1a7-a540-417b-aba5-dbcf54df1f04 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:36.433488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:36.434129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.434407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.434678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] PATH_INFO: `/v3/projects/d9cf766f238e4dcda649e0ae8a0c0926/users/fce84576e5f24a2c8dfbca3849c323d1/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.450218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9cf766f238e4dcda649e0ae8a0c0926, user_id=fce84576e5f24a2c8dfbca3849c323d1, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:36.452915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:36.466156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1302/2591] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:20:36 2022] PUT /identity/v3/projects/d9cf766f238e4dcda649e0ae8a0c0926/users/fce84576e5f24a2c8dfbca3849c323d1/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:36.476795 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652ef134-c2ff-43e2-bbda-51476fd87fba None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.477140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652ef134-c2ff-43e2-bbda-51476fd87fba None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.477418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652ef134-c2ff-43e2-bbda-51476fd87fba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.505547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-652ef134-c2ff-43e2-bbda-51476fd87fba None None] MFA Rules not processed for user `fce84576e5f24a2c8dfbca3849c323d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:36.507338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-652ef134-c2ff-43e2-bbda-51476fd87fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:36.571586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1290/2592] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:36.582589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:36.584344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:36.662006 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a25ec7-50a4-46a2-9681-ce57f914068b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:36.668440 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc6a785-fcb5-4da7-a58c-98901d104168 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:36.669040 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc6a785-fcb5-4da7-a58c-98901d104168 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:36.669316 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc6a785-fcb5-4da7-a58c-98901d104168 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:36.669587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc6a785-fcb5-4da7-a58c-98901d104168 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:36.677186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc6a785-fcb5-4da7-a58c-98901d104168 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:36.679822 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc6a785-fcb5-4da7-a58c-98901d104168 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:36.688637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1303/2593] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:20:45.013459 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652ef134-c2ff-43e2-bbda-51476fd87fba None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:45.074426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652ef134-c2ff-43e2-bbda-51476fd87fba None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:45.078799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:45.079416 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.079707 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.079993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] PATH_INFO: `/v3/users/fce84576e5f24a2c8dfbca3849c323d1` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.081127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] RBAC: Authorizing `identity:delete_user(user_id=fce84576e5f24a2c8dfbca3849c323d1)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:45.083337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:45.129429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fce84576e5f24a2c8dfbca3849c323d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.137192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fce84576e5f24a2c8dfbca3849c323d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.137811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fce84576e5f24a2c8dfbca3849c323d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.138427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fce84576e5f24a2c8dfbca3849c323d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.143014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:45.145958 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fce84576e5f24a2c8dfbca3849c323d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.166487 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1291/2594] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:20:45 2022] DELETE /identity/v3/users/fce84576e5f24a2c8dfbca3849c323d1 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:45.171161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca88d3d-91ef-49b5-8ce0-0ce8832c502e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.171663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca88d3d-91ef-49b5-8ce0-0ce8832c502e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.171783 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca88d3d-91ef-49b5-8ce0-0ce8832c502e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.190784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-bca88d3d-91ef-49b5-8ce0-0ce8832c502e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:45.192026 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bca88d3d-91ef-49b5-8ce0-0ce8832c502e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:45.232632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:45.253023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1304/2595] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:45.263162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:45.264625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:45.322777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2934b8f5-4bb2-48af-b38e-0e806521a78f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:45.326430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907acac7-47ad-4831-bea9-417032917f3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:45.326968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907acac7-47ad-4831-bea9-417032917f3f service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.327244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907acac7-47ad-4831-bea9-417032917f3f service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.327513 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907acac7-47ad-4831-bea9-417032917f3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.337290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907acac7-47ad-4831-bea9-417032917f3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:45.338698 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907acac7-47ad-4831-bea9-417032917f3f service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:45.347949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1292/2596] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:20:45.620970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca88d3d-91ef-49b5-8ce0-0ce8832c502e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:45.622470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bca88d3d-91ef-49b5-8ce0-0ce8832c502e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:45.634565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca88d3d-91ef-49b5-8ce0-0ce8832c502e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:45.639113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:45.639770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.640049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.640318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] PATH_INFO: `/v3/projects/d9cf766f238e4dcda649e0ae8a0c0926` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.641417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9cf766f238e4dcda649e0ae8a0c0926)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:45.643410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:45.690360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9cf766f238e4dcda649e0ae8a0c0926', 'request_id': 'req-8ea7e47c-5365-42aa-8080-4ebae6e6b950'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.698397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9cf766f238e4dcda649e0ae8a0c0926', 'request_id': 'req-8ea7e47c-5365-42aa-8080-4ebae6e6b950'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.704211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9cf766f238e4dcda649e0ae8a0c0926', 'request_id': 'req-8ea7e47c-5365-42aa-8080-4ebae6e6b950'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.715993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] The token cache is being invalidate because project d9cf766f238e4dcda649e0ae8a0c0926 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:20:45.716287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.716925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-8ea7e47c-5365-42aa-8080-4ebae6e6b950 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:45.718721 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1305/2597] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:20:45 2022] DELETE /identity/v3/projects/d9cf766f238e4dcda649e0ae8a0c0926 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:45.723254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a307c-506c-4d20-82d7-d40a997c8865 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.723536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a307c-506c-4d20-82d7-d40a997c8865 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.723837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a307c-506c-4d20-82d7-d40a997c8865 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.743867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c89a307c-506c-4d20-82d7-d40a997c8865 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:45.745066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c89a307c-506c-4d20-82d7-d40a997c8865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:45.804733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1293/2598] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:45.812728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b722f52e-a830-4e6c-a4cb-cf3c14f6063c None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.813012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b722f52e-a830-4e6c-a4cb-cf3c14f6063c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.813318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b722f52e-a830-4e6c-a4cb-cf3c14f6063c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.834014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-b722f52e-a830-4e6c-a4cb-cf3c14f6063c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:45.836082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b722f52e-a830-4e6c-a4cb-cf3c14f6063c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:45.843811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1306/2599] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:45.848212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a307c-506c-4d20-82d7-d40a997c8865 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:45.857855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a307c-506c-4d20-82d7-d40a997c8865 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:45.862326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:45.862913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.863189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.863458 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.864182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:20:45.864430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:45.866286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:45.877400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1294/2600] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:20:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:20:45.881994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b722f52e-a830-4e6c-a4cb-cf3c14f6063c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:45.891360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b722f52e-a830-4e6c-a4cb-cf3c14f6063c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:45.896108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dd9d1b-f4bd-4bd9-873a-20a3d57f374c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:45.896712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dd9d1b-f4bd-4bd9-873a-20a3d57f374c admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.897604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dd9d1b-f4bd-4bd9-873a-20a3d57f374c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.897881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dd9d1b-f4bd-4bd9-873a-20a3d57f374c admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.898540 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dd9d1b-f4bd-4bd9-873a-20a3d57f374c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:45.900494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dd9d1b-f4bd-4bd9-873a-20a3d57f374c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:45.927249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1307/2601] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:20:45 2022] POST /identity/v3/projects => generated 367 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:45.931479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:45.941233 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3469bec-165a-48c6-8968-6cde7be24fb3 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:45.996557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981d523-39d8-4681-898d-205413b445a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:45.997151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981d523-39d8-4681-898d-205413b445a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:45.997645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981d523-39d8-4681-898d-205413b445a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:45.997928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981d523-39d8-4681-898d-205413b445a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:45.998587 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7981d523-39d8-4681-898d-205413b445a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:46.000523 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7981d523-39d8-4681-898d-205413b445a2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:46.012171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:46.024007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1295/2602] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:20:45 2022] POST /identity/v3/users => generated 284 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:20:46.028099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dd9d1b-f4bd-4bd9-873a-20a3d57f374c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:46.038318 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dd9d1b-f4bd-4bd9-873a-20a3d57f374c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:46.042981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0c0317-5f95-422c-87d0-38ef063a9b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:46.043596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0c0317-5f95-422c-87d0-38ef063a9b14 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:46.043895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0c0317-5f95-422c-87d0-38ef063a9b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:46.044172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0c0317-5f95-422c-87d0-38ef063a9b14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:46.044814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0c0317-5f95-422c-87d0-38ef063a9b14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:46.046411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0c0317-5f95-422c-87d0-38ef063a9b14 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:46.056781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1308/2603] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:46.060635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981d523-39d8-4681-898d-205413b445a2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:46.071961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981d523-39d8-4681-898d-205413b445a2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:46.076572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aad1a87-9670-4667-823c-5c869fd29a28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:46.077151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad1a87-9670-4667-823c-5c869fd29a28 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:46.077430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad1a87-9670-4667-823c-5c869fd29a28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:46.077702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad1a87-9670-4667-823c-5c869fd29a28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:46.078340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aad1a87-9670-4667-823c-5c869fd29a28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:46.079954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aad1a87-9670-4667-823c-5c869fd29a28 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:46.090994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1296/2604] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:46.095526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0c0317-5f95-422c-87d0-38ef063a9b14 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:46.105573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0c0317-5f95-422c-87d0-38ef063a9b14 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:46.110180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:46.110823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:46.111107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:46.111384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] PATH_INFO: `/v3/projects/56db21745e7b4816aada8099dc2524ac/users/44d2adc9615c411dac99a4daf014104b/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:46.126596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56db21745e7b4816aada8099dc2524ac, user_id=44d2adc9615c411dac99a4daf014104b, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:46.129370 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:46.143048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1309/2605] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:20:46 2022] PUT /identity/v3/projects/56db21745e7b4816aada8099dc2524ac/users/44d2adc9615c411dac99a4daf014104b/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:46.154600 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c34e3-f249-45b8-a328-f5d229e0e860 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:46.154889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c34e3-f249-45b8-a328-f5d229e0e860 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:46.155170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c34e3-f249-45b8-a328-f5d229e0e860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:46.182899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d49c34e3-f249-45b8-a328-f5d229e0e860 None None] MFA Rules not processed for user `44d2adc9615c411dac99a4daf014104b`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:46.184145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d49c34e3-f249-45b8-a328-f5d229e0e860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:46.246906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1297/2606] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:46 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:46.258459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:46.260218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:46.316530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e564e80d-1bc4-43d6-8d80-5d0596c6c8b1 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:46.321099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64311f7-4d8f-4cf0-a4d1-0e10db5cdef0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:46.321698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64311f7-4d8f-4cf0-a4d1-0e10db5cdef0 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:46.321976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64311f7-4d8f-4cf0-a4d1-0e10db5cdef0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:46.322246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64311f7-4d8f-4cf0-a4d1-0e10db5cdef0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:46.330494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64311f7-4d8f-4cf0-a4d1-0e10db5cdef0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:46.331938 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64311f7-4d8f-4cf0-a4d1-0e10db5cdef0 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:46.340704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1310/2607] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:46 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:20:54.704476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c34e3-f249-45b8-a328-f5d229e0e860 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:54.765701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c34e3-f249-45b8-a328-f5d229e0e860 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:54.770220 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:54.770842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:54.771123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:54.771396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] PATH_INFO: `/v3/users/44d2adc9615c411dac99a4daf014104b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:54.772536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44d2adc9615c411dac99a4daf014104b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:54.774507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:54.818508 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44d2adc9615c411dac99a4daf014104b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:54.827239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44d2adc9615c411dac99a4daf014104b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:54.827889 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44d2adc9615c411dac99a4daf014104b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:54.828517 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44d2adc9615c411dac99a4daf014104b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:54.835844 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44d2adc9615c411dac99a4daf014104b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:54.845762 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:54.854268 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1298/2608] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:20:54 2022] DELETE /identity/v3/users/44d2adc9615c411dac99a4daf014104b => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:54.858760 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1e5518-412c-41dc-8f1d-6660542c91ef None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:54.859211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1e5518-412c-41dc-8f1d-6660542c91ef None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:54.859356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1e5518-412c-41dc-8f1d-6660542c91ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:54.873017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:54.879703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6c1e5518-412c-41dc-8f1d-6660542c91ef None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:54.881558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6c1e5518-412c-41dc-8f1d-6660542c91ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:54.941146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1311/2609] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:54.951860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:54.953251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:55.010318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663dd472-8297-4978-972c-c1e6df5270b3 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:55.014037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2068aa67-d992-4b59-87ed-e5b8c87fc67b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:55.014637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2068aa67-d992-4b59-87ed-e5b8c87fc67b service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.014916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2068aa67-d992-4b59-87ed-e5b8c87fc67b service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.015192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2068aa67-d992-4b59-87ed-e5b8c87fc67b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.024424 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2068aa67-d992-4b59-87ed-e5b8c87fc67b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:55.025843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2068aa67-d992-4b59-87ed-e5b8c87fc67b service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:55.035243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1299/2610] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:20:55.299771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1e5518-412c-41dc-8f1d-6660542c91ef None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:55.301218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6c1e5518-412c-41dc-8f1d-6660542c91ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:55.320422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1e5518-412c-41dc-8f1d-6660542c91ef None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:55.326114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:55.326756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.327041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.327321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] PATH_INFO: `/v3/projects/56db21745e7b4816aada8099dc2524ac` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.328486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=56db21745e7b4816aada8099dc2524ac)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:55.330416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:55.378025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56db21745e7b4816aada8099dc2524ac', 'request_id': 'req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:55.385284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56db21745e7b4816aada8099dc2524ac', 'request_id': 'req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:55.391587 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56db21745e7b4816aada8099dc2524ac', 'request_id': 'req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:55.403589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] The token cache is being invalidate because project 56db21745e7b4816aada8099dc2524ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:20:55.403910 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:55.404573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4385c962-5f2a-4d83-af6e-cf68cf43f6ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:55.406055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1312/2611] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:20:55 2022] DELETE /identity/v3/projects/56db21745e7b4816aada8099dc2524ac => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:55.410891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74791916-e222-4a72-ab4e-31f86fba0bda None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.411178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74791916-e222-4a72-ab4e-31f86fba0bda None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.411461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74791916-e222-4a72-ab4e-31f86fba0bda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.431583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-74791916-e222-4a72-ab4e-31f86fba0bda None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:55.432799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-74791916-e222-4a72-ab4e-31f86fba0bda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:55.491607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1300/2612] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:55.499972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039b42c-aedd-491d-8ee9-0d388f61e260 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.500257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039b42c-aedd-491d-8ee9-0d388f61e260 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.500528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8039b42c-aedd-491d-8ee9-0d388f61e260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.520013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-8039b42c-aedd-491d-8ee9-0d388f61e260 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:55.521203 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8039b42c-aedd-491d-8ee9-0d388f61e260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:55.529828 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1313/2613] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:55.534178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74791916-e222-4a72-ab4e-31f86fba0bda None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:55.544382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74791916-e222-4a72-ab4e-31f86fba0bda None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:55.548985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:55.549591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.549871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.550146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.550827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:20:55.551130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:55.552775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:55.564910 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1301/2614] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:20:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:20:55.569138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8039b42c-aedd-491d-8ee9-0d388f61e260 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:55.578531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8039b42c-aedd-491d-8ee9-0d388f61e260 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:55.583842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2954f4a7-db06-47f9-b76a-53683cd63298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:55.584432 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2954f4a7-db06-47f9-b76a-53683cd63298 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.584711 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2954f4a7-db06-47f9-b76a-53683cd63298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.584981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2954f4a7-db06-47f9-b76a-53683cd63298 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.588082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2954f4a7-db06-47f9-b76a-53683cd63298 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:55.588082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2954f4a7-db06-47f9-b76a-53683cd63298 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:55.614717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1314/2615] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:20:55 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:55.618920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:55.628841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686c8ee8-a2b8-465a-8fdb-6e220cbecc91 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:55.685281 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57af20-e8d7-4ca3-8e25-c64823a3a22e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:55.685872 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57af20-e8d7-4ca3-8e25-c64823a3a22e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.686148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57af20-e8d7-4ca3-8e25-c64823a3a22e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.686418 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57af20-e8d7-4ca3-8e25-c64823a3a22e admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.687046 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c57af20-e8d7-4ca3-8e25-c64823a3a22e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:55.688979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c57af20-e8d7-4ca3-8e25-c64823a3a22e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:55.709571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1302/2616] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:20:55 2022] POST /identity/v3/users => generated 286 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:55.714182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2954f4a7-db06-47f9-b76a-53683cd63298 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:55.724444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2954f4a7-db06-47f9-b76a-53683cd63298 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:55.729883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935571d1-fe7b-42a3-bf5e-7c8365f51193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:55.730477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935571d1-fe7b-42a3-bf5e-7c8365f51193 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.730759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935571d1-fe7b-42a3-bf5e-7c8365f51193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.731036 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935571d1-fe7b-42a3-bf5e-7c8365f51193 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.731677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935571d1-fe7b-42a3-bf5e-7c8365f51193 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:55.733306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935571d1-fe7b-42a3-bf5e-7c8365f51193 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:55.743630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1315/2617] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:55.747310 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57af20-e8d7-4ca3-8e25-c64823a3a22e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:55.757643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57af20-e8d7-4ca3-8e25-c64823a3a22e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:55.762284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaacdb1e-b7b9-439b-acbf-9ee6a9f6716c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:55.762884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacdb1e-b7b9-439b-acbf-9ee6a9f6716c admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.763193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacdb1e-b7b9-439b-acbf-9ee6a9f6716c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.763471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacdb1e-b7b9-439b-acbf-9ee6a9f6716c admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.764149 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaacdb1e-b7b9-439b-acbf-9ee6a9f6716c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:55.765741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaacdb1e-b7b9-439b-acbf-9ee6a9f6716c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:55.776161 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1303/2618] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:20:55.780334 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935571d1-fe7b-42a3-bf5e-7c8365f51193 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:55.791076 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935571d1-fe7b-42a3-bf5e-7c8365f51193 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:55.795709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfed9861-1bd8-4a0a-afae-da6455e0fbfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:55.796370 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfed9861-1bd8-4a0a-afae-da6455e0fbfb admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.796677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfed9861-1bd8-4a0a-afae-da6455e0fbfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.796956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfed9861-1bd8-4a0a-afae-da6455e0fbfb admin admin] PATH_INFO: `/v3/projects/b40392b9b60340df846d6311c47234ba/users/85398c1b3e1d47f592c8ac3a519dd028/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.811533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfed9861-1bd8-4a0a-afae-da6455e0fbfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b40392b9b60340df846d6311c47234ba, user_id=85398c1b3e1d47f592c8ac3a519dd028, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:55.814273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfed9861-1bd8-4a0a-afae-da6455e0fbfb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:55.827957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1316/2619] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:20:55 2022] PUT /identity/v3/projects/b40392b9b60340df846d6311c47234ba/users/85398c1b3e1d47f592c8ac3a519dd028/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:55.840166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caf673f-c1c9-452d-bed6-b1e74b36af5d None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:55.840446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caf673f-c1c9-452d-bed6-b1e74b36af5d None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:55.840717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caf673f-c1c9-452d-bed6-b1e74b36af5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:55.847778 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:55.871454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-0caf673f-c1c9-452d-bed6-b1e74b36af5d None None] MFA Rules not processed for user `85398c1b3e1d47f592c8ac3a519dd028`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:55.872711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0caf673f-c1c9-452d-bed6-b1e74b36af5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:55.936607 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1304/2620] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:55 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:55.941113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfed9861-1bd8-4a0a-afae-da6455e0fbfb admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:55.971905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:56.005299 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfed9861-1bd8-4a0a-afae-da6455e0fbfb admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:56.009990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b543fa93-0d62-4d44-b7c3-b520d9446ea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:56.010584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543fa93-0d62-4d44-b7c3-b520d9446ea9 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:56.010860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543fa93-0d62-4d44-b7c3-b520d9446ea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:56.011127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543fa93-0d62-4d44-b7c3-b520d9446ea9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:56.011788 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b543fa93-0d62-4d44-b7c3-b520d9446ea9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:56.013675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b543fa93-0d62-4d44-b7c3-b520d9446ea9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:56.039790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1317/2621] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:20:55 2022] POST /identity/v3/projects => generated 373 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:56.043921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caf673f-c1c9-452d-bed6-b1e74b36af5d None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:56.054949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caf673f-c1c9-452d-bed6-b1e74b36af5d None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:56.110402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3debbe1-f64f-4043-bd87-1f4e4f12a6d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:56.111058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3debbe1-f64f-4043-bd87-1f4e4f12a6d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:56.111337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3debbe1-f64f-4043-bd87-1f4e4f12a6d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:56.111633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3debbe1-f64f-4043-bd87-1f4e4f12a6d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:56.112298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3debbe1-f64f-4043-bd87-1f4e4f12a6d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:56.114221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3debbe1-f64f-4043-bd87-1f4e4f12a6d3 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:56.135774 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1305/2622] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:20:56 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:56.139794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b543fa93-0d62-4d44-b7c3-b520d9446ea9 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:56.151036 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b543fa93-0d62-4d44-b7c3-b520d9446ea9 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:56.155586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777d02eb-cd5b-46a5-9067-158974311b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:56.156209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777d02eb-cd5b-46a5-9067-158974311b35 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:56.156493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777d02eb-cd5b-46a5-9067-158974311b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:56.156767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777d02eb-cd5b-46a5-9067-158974311b35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:56.157406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777d02eb-cd5b-46a5-9067-158974311b35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:56.159166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777d02eb-cd5b-46a5-9067-158974311b35 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:56.169507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1318/2623] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:56.173550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3debbe1-f64f-4043-bd87-1f4e4f12a6d3 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:56.184174 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3debbe1-f64f-4043-bd87-1f4e4f12a6d3 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:56.188683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c20503e-73c8-4661-a883-48d79507337a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:56.189314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c20503e-73c8-4661-a883-48d79507337a admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:56.189588 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c20503e-73c8-4661-a883-48d79507337a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:56.189855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c20503e-73c8-4661-a883-48d79507337a admin admin] PATH_INFO: `/v3/projects/73b5fb15769c4dcb81263af531b2f53d/users/43f5a1f93ba94b2cb6619bb86eb452f7/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:56.205376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c20503e-73c8-4661-a883-48d79507337a admin admin] RBAC: Authorizing `identity:create_grant(project_id=73b5fb15769c4dcb81263af531b2f53d, user_id=43f5a1f93ba94b2cb6619bb86eb452f7, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:56.207994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c20503e-73c8-4661-a883-48d79507337a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:56.221087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1306/2624] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:20:56 2022] PUT /identity/v3/projects/73b5fb15769c4dcb81263af531b2f53d/users/43f5a1f93ba94b2cb6619bb86eb452f7/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:56.224944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777d02eb-cd5b-46a5-9067-158974311b35 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:56.236042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777d02eb-cd5b-46a5-9067-158974311b35 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:56.292466 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8d85ba-c466-4a8d-8a4e-9caf8505bbfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:56.292997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8d85ba-c466-4a8d-8a4e-9caf8505bbfb admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:56.293279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8d85ba-c466-4a8d-8a4e-9caf8505bbfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:56.293552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8d85ba-c466-4a8d-8a4e-9caf8505bbfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:56.294180 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8d85ba-c466-4a8d-8a4e-9caf8505bbfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:56.295852 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8d85ba-c466-4a8d-8a4e-9caf8505bbfb admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:56.306215 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1319/2625] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:56 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:56.310394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c20503e-73c8-4661-a883-48d79507337a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:56.321859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c20503e-73c8-4661-a883-48d79507337a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:56.326710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:56.327364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:56.327644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:56.327953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] PATH_INFO: `/v3/projects/73b5fb15769c4dcb81263af531b2f53d/users/43f5a1f93ba94b2cb6619bb86eb452f7/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:56.329878 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] RBAC: Authorizing `identity:create_grant(project_id=73b5fb15769c4dcb81263af531b2f53d, user_id=43f5a1f93ba94b2cb6619bb86eb452f7, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:56.332558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:56.343328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1307/2626] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:20:56 2022] PUT /identity/v3/projects/73b5fb15769c4dcb81263af531b2f53d/users/43f5a1f93ba94b2cb6619bb86eb452f7/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:56.353607 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96c8f15-9138-4006-ae6e-c360452992f1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:56.353993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96c8f15-9138-4006-ae6e-c360452992f1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:56.354333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96c8f15-9138-4006-ae6e-c360452992f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:56.382560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-f96c8f15-9138-4006-ae6e-c360452992f1 None None] MFA Rules not processed for user `43f5a1f93ba94b2cb6619bb86eb452f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:56.384189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f96c8f15-9138-4006-ae6e-c360452992f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:56.452093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1320/2627] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:56 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:56.463388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:56.465091 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:56.511926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:56.523401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fadcb09-1e50-4f5a-945f-3f2846cb086d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:56.527028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72ea2a5-a940-40eb-9fe0-c8817420f2cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:56.527622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72ea2a5-a940-40eb-9fe0-c8817420f2cb service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:56.527950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72ea2a5-a940-40eb-9fe0-c8817420f2cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:56.528223 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72ea2a5-a940-40eb-9fe0-c8817420f2cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:56.537060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72ea2a5-a940-40eb-9fe0-c8817420f2cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:56.539183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72ea2a5-a940-40eb-9fe0-c8817420f2cb service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:56.548538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1308/2628] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:56 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:20:56.932377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96c8f15-9138-4006-ae6e-c360452992f1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:56.957945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:56.996294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96c8f15-9138-4006-ae6e-c360452992f1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:57.000810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:57.001436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:57.001718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:57.001993 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] PATH_INFO: `/v3/users/85398c1b3e1d47f592c8ac3a519dd028` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:57.003096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=85398c1b3e1d47f592c8ac3a519dd028)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:57.005064 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:57.050616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85398c1b3e1d47f592c8ac3a519dd028'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.058407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85398c1b3e1d47f592c8ac3a519dd028'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.059022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85398c1b3e1d47f592c8ac3a519dd028'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.059644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85398c1b3e1d47f592c8ac3a519dd028'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.067215 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85398c1b3e1d47f592c8ac3a519dd028'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.085459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1321/2629] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:20:56 2022] DELETE /identity/v3/users/85398c1b3e1d47f592c8ac3a519dd028 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:57.090230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d0e3cb-6ea2-4adb-8643-3604f64b96b1 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:57.090687 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d0e3cb-6ea2-4adb-8643-3604f64b96b1 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:57.090782 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d0e3cb-6ea2-4adb-8643-3604f64b96b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:57.109578 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-78d0e3cb-6ea2-4adb-8643-3604f64b96b1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:57.110777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-78d0e3cb-6ea2-4adb-8643-3604f64b96b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:57.168253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1309/2630] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:57.177923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:57.180013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:57.236769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0d6b96-3a72-4b21-9698-4148e4e584c3 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:57.240444 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468368d5-56df-4fae-9ddb-c0d6c0476622 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:57.241042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468368d5-56df-4fae-9ddb-c0d6c0476622 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:57.242016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468368d5-56df-4fae-9ddb-c0d6c0476622 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:57.242403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468368d5-56df-4fae-9ddb-c0d6c0476622 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:57.251319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468368d5-56df-4fae-9ddb-c0d6c0476622 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:57.252777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468368d5-56df-4fae-9ddb-c0d6c0476622 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:57.262703 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1322/2631] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:20:57.529117 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d0e3cb-6ea2-4adb-8643-3604f64b96b1 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:57.530557 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-78d0e3cb-6ea2-4adb-8643-3604f64b96b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:57.545292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d0e3cb-6ea2-4adb-8643-3604f64b96b1 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:57.551400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:57.552906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:57.553291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:57.553704 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] PATH_INFO: `/v3/projects/b40392b9b60340df846d6311c47234ba` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:57.555387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b40392b9b60340df846d6311c47234ba)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:57.557332 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:57.603679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b40392b9b60340df846d6311c47234ba', 'request_id': 'req-bfadf306-dece-4301-bcf4-7a637f12ab41'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.610493 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b40392b9b60340df846d6311c47234ba', 'request_id': 'req-bfadf306-dece-4301-bcf4-7a637f12ab41'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.616051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b40392b9b60340df846d6311c47234ba', 'request_id': 'req-bfadf306-dece-4301-bcf4-7a637f12ab41'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.627776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] The token cache is being invalidate because project b40392b9b60340df846d6311c47234ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:20:57.628060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.628695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.630153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1310/2632] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:20:57 2022] DELETE /identity/v3/projects/b40392b9b60340df846d6311c47234ba => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:57.634116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468368d5-56df-4fae-9ddb-c0d6c0476622 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:57.643497 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468368d5-56df-4fae-9ddb-c0d6c0476622 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:57.685200 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:57.701882 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:57.702604 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:57.702962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:57.703272 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] PATH_INFO: `/v3/users/43f5a1f93ba94b2cb6619bb86eb452f7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:57.704435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43f5a1f93ba94b2cb6619bb86eb452f7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:57.706470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:57.751093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43f5a1f93ba94b2cb6619bb86eb452f7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.759132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43f5a1f93ba94b2cb6619bb86eb452f7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.759774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43f5a1f93ba94b2cb6619bb86eb452f7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.760379 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43f5a1f93ba94b2cb6619bb86eb452f7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.767090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6bc619ff-852d-441e-934b-770022567750 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43f5a1f93ba94b2cb6619bb86eb452f7'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:57.784514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1323/2633] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:20:57 2022] DELETE /identity/v3/users/43f5a1f93ba94b2cb6619bb86eb452f7 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:58.054285 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.055809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:58.068956 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfadf306-dece-4301-bcf4-7a637f12ab41 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.125674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.126339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.126567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.126839 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] PATH_INFO: `/v3/projects/73b5fb15769c4dcb81263af531b2f53d` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.128018 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=73b5fb15769c4dcb81263af531b2f53d)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.129865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.141393 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:58.177243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73b5fb15769c4dcb81263af531b2f53d', 'request_id': 'req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:58.184253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73b5fb15769c4dcb81263af531b2f53d', 'request_id': 'req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:58.189674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73b5fb15769c4dcb81263af531b2f53d', 'request_id': 'req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:58.201292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] The token cache is being invalidate because project 73b5fb15769c4dcb81263af531b2f53d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:20:58.201579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:58.202196 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a49eb201-f999-4fae-84c5-fd3d9c1dfc17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:20:58.203665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1311/2634] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:20:58 2022] DELETE /identity/v3/projects/73b5fb15769c4dcb81263af531b2f53d => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:58.208582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147c8efc-db1a-41ab-afd6-fe01b4aa6c9b None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.209051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147c8efc-db1a-41ab-afd6-fe01b4aa6c9b None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.209157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147c8efc-db1a-41ab-afd6-fe01b4aa6c9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.229119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-147c8efc-db1a-41ab-afd6-fe01b4aa6c9b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:58.230327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-147c8efc-db1a-41ab-afd6-fe01b4aa6c9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:58.287615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1324/2635] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:58.296075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6566d6-8810-4a46-8149-b9fb03480f70 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.296382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6566d6-8810-4a46-8149-b9fb03480f70 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.296635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6566d6-8810-4a46-8149-b9fb03480f70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.317258 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-9b6566d6-8810-4a46-8149-b9fb03480f70 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:58.318465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9b6566d6-8810-4a46-8149-b9fb03480f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:58.326765 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1312/2636] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:58.331096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147c8efc-db1a-41ab-afd6-fe01b4aa6c9b None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.341254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147c8efc-db1a-41ab-afd6-fe01b4aa6c9b None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.345867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.346464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.346746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.347139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.347830 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:20:58.348139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.349744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.361786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1325/2637] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:20:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:20:58.365954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6566d6-8810-4a46-8149-b9fb03480f70 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.375851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6566d6-8810-4a46-8149-b9fb03480f70 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.380522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d05562e-a7fe-4c20-8aa1-561ee0028b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.381081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d05562e-a7fe-4c20-8aa1-561ee0028b20 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.381357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d05562e-a7fe-4c20-8aa1-561ee0028b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.381628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d05562e-a7fe-4c20-8aa1-561ee0028b20 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.382342 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d05562e-a7fe-4c20-8aa1-561ee0028b20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.384228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d05562e-a7fe-4c20-8aa1-561ee0028b20 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.411663 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1313/2638] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:20:58 2022] POST /identity/v3/projects => generated 365 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:58.416924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.426810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f051d750-e61b-485f-bb28-7f7b1e625f09 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.477999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:58.489519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a36afa-54ef-4ff8-a594-00ab74444641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.490126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a36afa-54ef-4ff8-a594-00ab74444641 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.490404 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a36afa-54ef-4ff8-a594-00ab74444641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.490676 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a36afa-54ef-4ff8-a594-00ab74444641 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.491298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a36afa-54ef-4ff8-a594-00ab74444641 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.493271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a36afa-54ef-4ff8-a594-00ab74444641 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.513840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1326/2639] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:20:58 2022] POST /identity/v3/users => generated 283 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:58.518037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d05562e-a7fe-4c20-8aa1-561ee0028b20 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.528883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d05562e-a7fe-4c20-8aa1-561ee0028b20 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.533438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebca845f-2132-4abe-8079-0e40adf32640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.534047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebca845f-2132-4abe-8079-0e40adf32640 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.534314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebca845f-2132-4abe-8079-0e40adf32640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.534591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebca845f-2132-4abe-8079-0e40adf32640 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.535236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebca845f-2132-4abe-8079-0e40adf32640 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.536864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebca845f-2132-4abe-8079-0e40adf32640 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.547232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1314/2640] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:58.551184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a36afa-54ef-4ff8-a594-00ab74444641 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.564472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a36afa-54ef-4ff8-a594-00ab74444641 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.569482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48b41e2-eef0-4e9c-bbac-a0e4820b1547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.570147 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b41e2-eef0-4e9c-bbac-a0e4820b1547 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.570427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b41e2-eef0-4e9c-bbac-a0e4820b1547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.570965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b41e2-eef0-4e9c-bbac-a0e4820b1547 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.571653 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48b41e2-eef0-4e9c-bbac-a0e4820b1547 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.573317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48b41e2-eef0-4e9c-bbac-a0e4820b1547 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.584763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1327/2641] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:58.589581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebca845f-2132-4abe-8079-0e40adf32640 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.599564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebca845f-2132-4abe-8079-0e40adf32640 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.605489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607a7f0f-2a5d-4b3f-a9d7-64119115325f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.606144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a7f0f-2a5d-4b3f-a9d7-64119115325f admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.606422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a7f0f-2a5d-4b3f-a9d7-64119115325f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.606691 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a7f0f-2a5d-4b3f-a9d7-64119115325f admin admin] PATH_INFO: `/v3/projects/54a7756500ea4f4597a4bdf063acd346/users/aafc0830f0164576aae7ad1d250d3d6b/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.624346 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607a7f0f-2a5d-4b3f-a9d7-64119115325f admin admin] RBAC: Authorizing `identity:create_grant(project_id=54a7756500ea4f4597a4bdf063acd346, user_id=aafc0830f0164576aae7ad1d250d3d6b, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.627052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607a7f0f-2a5d-4b3f-a9d7-64119115325f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.641062 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1315/2642] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:20:58 2022] PUT /identity/v3/projects/54a7756500ea4f4597a4bdf063acd346/users/aafc0830f0164576aae7ad1d250d3d6b/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:58.653187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcde2c8-3d9b-452d-a33c-f3af0dcad83e None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.653475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcde2c8-3d9b-452d-a33c-f3af0dcad83e None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.653753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcde2c8-3d9b-452d-a33c-f3af0dcad83e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.682504 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-5fcde2c8-3d9b-452d-a33c-f3af0dcad83e None None] MFA Rules not processed for user `aafc0830f0164576aae7ad1d250d3d6b`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:58.683766 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5fcde2c8-3d9b-452d-a33c-f3af0dcad83e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:58.749368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1328/2643] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:58 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:58.753714 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607a7f0f-2a5d-4b3f-a9d7-64119115325f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.817413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607a7f0f-2a5d-4b3f-a9d7-64119115325f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.824209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dc7301-b0c2-4c17-a962-dd5d3f35124e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.824887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dc7301-b0c2-4c17-a962-dd5d3f35124e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.824997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dc7301-b0c2-4c17-a962-dd5d3f35124e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.825206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dc7301-b0c2-4c17-a962-dd5d3f35124e admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.825829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dc7301-b0c2-4c17-a962-dd5d3f35124e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.827756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dc7301-b0c2-4c17-a962-dd5d3f35124e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.854922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1316/2644] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:20:58 2022] POST /identity/v3/projects => generated 365 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:58.859189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcde2c8-3d9b-452d-a33c-f3af0dcad83e None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.870895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcde2c8-3d9b-452d-a33c-f3af0dcad83e None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.926145 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f168f2d9-a25a-4c42-9a01-ed8bd9d6587d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.926673 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168f2d9-a25a-4c42-9a01-ed8bd9d6587d admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.926950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168f2d9-a25a-4c42-9a01-ed8bd9d6587d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.927222 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168f2d9-a25a-4c42-9a01-ed8bd9d6587d admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.927877 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f168f2d9-a25a-4c42-9a01-ed8bd9d6587d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.929864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f168f2d9-a25a-4c42-9a01-ed8bd9d6587d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.951456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1329/2645] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:20:58 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:20:58.955507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dc7301-b0c2-4c17-a962-dd5d3f35124e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:58.966611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dc7301-b0c2-4c17-a962-dd5d3f35124e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:58.971157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ca9cd3-de95-4bcd-97b9-ac6036772acc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:58.971780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ca9cd3-de95-4bcd-97b9-ac6036772acc admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:58.972063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ca9cd3-de95-4bcd-97b9-ac6036772acc admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:58.972336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ca9cd3-de95-4bcd-97b9-ac6036772acc admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:58.972997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ca9cd3-de95-4bcd-97b9-ac6036772acc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:58.974611 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ca9cd3-de95-4bcd-97b9-ac6036772acc admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:58.985635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1317/2646] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:20:58.989697 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f168f2d9-a25a-4c42-9a01-ed8bd9d6587d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:59.000553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f168f2d9-a25a-4c42-9a01-ed8bd9d6587d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:59.005159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb5ae69-9abb-484c-b435-5702b85ad772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:59.005863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb5ae69-9abb-484c-b435-5702b85ad772 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:59.006290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb5ae69-9abb-484c-b435-5702b85ad772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:59.006614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb5ae69-9abb-484c-b435-5702b85ad772 admin admin] PATH_INFO: `/v3/projects/9c1dc778904e4fe88cda24fe9d33a452/users/875b68ff693d42f78862bcc49be2ca57/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:59.021275 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb5ae69-9abb-484c-b435-5702b85ad772 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c1dc778904e4fe88cda24fe9d33a452, user_id=875b68ff693d42f78862bcc49be2ca57, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:59.024175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb5ae69-9abb-484c-b435-5702b85ad772 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:59.037249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1330/2647] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:20:58 2022] PUT /identity/v3/projects/9c1dc778904e4fe88cda24fe9d33a452/users/875b68ff693d42f78862bcc49be2ca57/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:59.042099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ca9cd3-de95-4bcd-97b9-ac6036772acc admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:59.051993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ca9cd3-de95-4bcd-97b9-ac6036772acc admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:59.060595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:59.111760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22b9c34-62d4-48d4-b619-69745ff63fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:59.112363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b9c34-62d4-48d4-b619-69745ff63fa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:59.112643 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b9c34-62d4-48d4-b619-69745ff63fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:59.112944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22b9c34-62d4-48d4-b619-69745ff63fa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:59.113579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22b9c34-62d4-48d4-b619-69745ff63fa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:59.115197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22b9c34-62d4-48d4-b619-69745ff63fa4 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:59.125647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1318/2648] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:20:59 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:20:59.129903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb5ae69-9abb-484c-b435-5702b85ad772 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:59.140148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb5ae69-9abb-484c-b435-5702b85ad772 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:59.144709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:59.145343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:59.145617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:59.145886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] PATH_INFO: `/v3/projects/9c1dc778904e4fe88cda24fe9d33a452/users/875b68ff693d42f78862bcc49be2ca57/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:59.148820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c1dc778904e4fe88cda24fe9d33a452, user_id=875b68ff693d42f78862bcc49be2ca57, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:59.151666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:59.161844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1331/2649] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:20:59 2022] PUT /identity/v3/projects/9c1dc778904e4fe88cda24fe9d33a452/users/875b68ff693d42f78862bcc49be2ca57/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:20:59.172536 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee2576-9f0c-45be-b0f2-492c0213b61e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:59.172994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee2576-9f0c-45be-b0f2-492c0213b61e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:59.173105 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee2576-9f0c-45be-b0f2-492c0213b61e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:59.200692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c8ee2576-9f0c-45be-b0f2-492c0213b61e None None] MFA Rules not processed for user `875b68ff693d42f78862bcc49be2ca57`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:20:59.201895 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c8ee2576-9f0c-45be-b0f2-492c0213b61e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:59.268382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1319/2650] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:20:59 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:20:59.280397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:59.282113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:20:59.296594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:20:59.341749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22abab8f-3f5b-435b-9f99-6c92be59c8de admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:59.345414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c5ee74-c03c-45fa-bd09-c99162e31a4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:59.346016 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5ee74-c03c-45fa-bd09-c99162e31a4a service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:59.346297 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5ee74-c03c-45fa-bd09-c99162e31a4a service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:59.346572 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c5ee74-c03c-45fa-bd09-c99162e31a4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:59.356045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c5ee74-c03c-45fa-bd09-c99162e31a4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:59.357478 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c5ee74-c03c-45fa-bd09-c99162e31a4a service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:59.366823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1332/2651] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:59 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:20:59.443125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee2576-9f0c-45be-b0f2-492c0213b61e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:20:59.453200 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee2576-9f0c-45be-b0f2-492c0213b61e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:20:59.456772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796d636e-372e-4345-bb57-500e468491c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:20:59.457366 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796d636e-372e-4345-bb57-500e468491c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:20:59.457647 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796d636e-372e-4345-bb57-500e468491c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:20:59.457921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796d636e-372e-4345-bb57-500e468491c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:20:59.513660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796d636e-372e-4345-bb57-500e468491c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:20:59.515088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796d636e-372e-4345-bb57-500e468491c5 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:20:59.524679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1320/2652] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:20:59 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:02.245300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c5ee74-c03c-45fa-bd09-c99162e31a4a service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:02.307287 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c5ee74-c03c-45fa-bd09-c99162e31a4a service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:02.311888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:02.312519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:02.312798 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:02.313072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] PATH_INFO: `/v3/users/aafc0830f0164576aae7ad1d250d3d6b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:02.314185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=aafc0830f0164576aae7ad1d250d3d6b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:02.316160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:02.363013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aafc0830f0164576aae7ad1d250d3d6b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.371752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aafc0830f0164576aae7ad1d250d3d6b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.372401 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aafc0830f0164576aae7ad1d250d3d6b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.373027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aafc0830f0164576aae7ad1d250d3d6b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.380117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aafc0830f0164576aae7ad1d250d3d6b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.398326 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1333/2653] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:21:02 2022] DELETE /identity/v3/users/aafc0830f0164576aae7ad1d250d3d6b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:02.403232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e4171-b38e-4828-94e3-20e85cbb4a93 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:02.403740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e4171-b38e-4828-94e3-20e85cbb4a93 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:02.403858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e4171-b38e-4828-94e3-20e85cbb4a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:02.423596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f08e4171-b38e-4828-94e3-20e85cbb4a93 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:02.424933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f08e4171-b38e-4828-94e3-20e85cbb4a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:02.451822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:02.485320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1321/2654] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:02.495770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:02.497187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:02.532288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:02.555785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16789eef-cdb1-45ee-929e-60007d4754dd admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:02.560124 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e06d14-be7d-42b4-b416-130b976dfaf3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:02.560662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e06d14-be7d-42b4-b416-130b976dfaf3 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:02.560941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e06d14-be7d-42b4-b416-130b976dfaf3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:02.561213 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e06d14-be7d-42b4-b416-130b976dfaf3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:02.570138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e06d14-be7d-42b4-b416-130b976dfaf3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:02.571863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e06d14-be7d-42b4-b416-130b976dfaf3 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:02.581230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1334/2655] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:02.862514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08e4171-b38e-4828-94e3-20e85cbb4a93 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:02.863980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f08e4171-b38e-4828-94e3-20e85cbb4a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:02.880017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08e4171-b38e-4828-94e3-20e85cbb4a93 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:02.886696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:02.887352 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:02.887627 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:02.887924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] PATH_INFO: `/v3/projects/54a7756500ea4f4597a4bdf063acd346` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:02.889060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] RBAC: Authorizing `identity:delete_project(project_id=54a7756500ea4f4597a4bdf063acd346)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:02.891007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:02.947828 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54a7756500ea4f4597a4bdf063acd346', 'request_id': 'req-693088da-d566-40e9-b77c-de56d3c21183'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.955881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54a7756500ea4f4597a4bdf063acd346', 'request_id': 'req-693088da-d566-40e9-b77c-de56d3c21183'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.962263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54a7756500ea4f4597a4bdf063acd346', 'request_id': 'req-693088da-d566-40e9-b77c-de56d3c21183'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.976807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] The token cache is being invalidate because project 54a7756500ea4f4597a4bdf063acd346 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:21:02.977098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.978118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:02.980716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1322/2656] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:21:02 2022] DELETE /identity/v3/projects/54a7756500ea4f4597a4bdf063acd346 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:02.984839 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e06d14-be7d-42b4-b416-130b976dfaf3 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:02.995552 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e06d14-be7d-42b4-b416-130b976dfaf3 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:03.060321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:03.060981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:03.061258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:03.061602 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] PATH_INFO: `/v3/users/875b68ff693d42f78862bcc49be2ca57` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:03.064025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] RBAC: Authorizing `identity:delete_user(user_id=875b68ff693d42f78862bcc49be2ca57)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:03.066323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:03.124365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '875b68ff693d42f78862bcc49be2ca57'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.133702 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '875b68ff693d42f78862bcc49be2ca57'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.134327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '875b68ff693d42f78862bcc49be2ca57'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.134966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '875b68ff693d42f78862bcc49be2ca57'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.142218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-68165be1-be1c-4c16-8043-18d37609ab98 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '875b68ff693d42f78862bcc49be2ca57'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.165561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1335/2657] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:21:02 2022] DELETE /identity/v3/users/875b68ff693d42f78862bcc49be2ca57 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:03.483901 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:03.485441 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:03.501491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693088da-d566-40e9-b77c-de56d3c21183 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:03.565711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:03.566339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:03.566625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:03.566916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] PATH_INFO: `/v3/projects/9c1dc778904e4fe88cda24fe9d33a452` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:03.568049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c1dc778904e4fe88cda24fe9d33a452)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:03.569942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:03.618051 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c1dc778904e4fe88cda24fe9d33a452', 'request_id': 'req-863f5144-28fd-4e27-87dd-d3b24b8e386d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.626068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c1dc778904e4fe88cda24fe9d33a452', 'request_id': 'req-863f5144-28fd-4e27-87dd-d3b24b8e386d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.627401 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:03.634274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c1dc778904e4fe88cda24fe9d33a452', 'request_id': 'req-863f5144-28fd-4e27-87dd-d3b24b8e386d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.647109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] The token cache is being invalidate because project 9c1dc778904e4fe88cda24fe9d33a452 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:21:03.647328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.647995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-863f5144-28fd-4e27-87dd-d3b24b8e386d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:03.649465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1323/2658] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:21:03 2022] DELETE /identity/v3/projects/9c1dc778904e4fe88cda24fe9d33a452 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:03.654484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3826c595-5891-4d6c-a07c-b30f528ebb6a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:03.654966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3826c595-5891-4d6c-a07c-b30f528ebb6a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:03.655074 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3826c595-5891-4d6c-a07c-b30f528ebb6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:03.674149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-3826c595-5891-4d6c-a07c-b30f528ebb6a None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:03.675345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3826c595-5891-4d6c-a07c-b30f528ebb6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:03.723315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:03.735638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1336/2659] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:03.745344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94a82ee-7a62-450d-9954-11cc84cbb72f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:03.745630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94a82ee-7a62-450d-9954-11cc84cbb72f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:03.745904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94a82ee-7a62-450d-9954-11cc84cbb72f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:03.765867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c94a82ee-7a62-450d-9954-11cc84cbb72f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:03.767078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c94a82ee-7a62-450d-9954-11cc84cbb72f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:03.775229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1324/2660] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:03.779585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3826c595-5891-4d6c-a07c-b30f528ebb6a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:03.789315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3826c595-5891-4d6c-a07c-b30f528ebb6a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:03.794211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:03.794864 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:03.795144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:03.795419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:03.796103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:21:03.796411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:03.798041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:03.810082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1337/2661] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:21:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:21:03.814317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94a82ee-7a62-450d-9954-11cc84cbb72f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:03.824511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94a82ee-7a62-450d-9954-11cc84cbb72f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:03.829750 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138dd765-1cf5-4fb6-bde8-246de9e9bb93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:03.830354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138dd765-1cf5-4fb6-bde8-246de9e9bb93 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:03.830633 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138dd765-1cf5-4fb6-bde8-246de9e9bb93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:03.830907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138dd765-1cf5-4fb6-bde8-246de9e9bb93 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:03.831569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138dd765-1cf5-4fb6-bde8-246de9e9bb93 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:03.833572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138dd765-1cf5-4fb6-bde8-246de9e9bb93 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:03.860348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1325/2662] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:21:03 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:03.864636 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:03.875363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612e4dce-2df1-4514-93f1-ef0d27375943 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:03.931486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67828fe0-6c5f-47c7-864a-d658957f365c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:03.932792 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67828fe0-6c5f-47c7-864a-d658957f365c admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:03.933086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67828fe0-6c5f-47c7-864a-d658957f365c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:03.933360 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67828fe0-6c5f-47c7-864a-d658957f365c admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:03.933997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67828fe0-6c5f-47c7-864a-d658957f365c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:03.935945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67828fe0-6c5f-47c7-864a-d658957f365c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:03.960506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1338/2663] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:21:03 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:03.962213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138dd765-1cf5-4fb6-bde8-246de9e9bb93 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:03.972628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138dd765-1cf5-4fb6-bde8-246de9e9bb93 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:03.977178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1760f4e4-1e84-4ffa-aa57-eeb8ce7aceb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:03.977766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1760f4e4-1e84-4ffa-aa57-eeb8ce7aceb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:03.978041 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1760f4e4-1e84-4ffa-aa57-eeb8ce7aceb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:03.978308 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1760f4e4-1e84-4ffa-aa57-eeb8ce7aceb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:03.979461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1760f4e4-1e84-4ffa-aa57-eeb8ce7aceb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:03.981065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1760f4e4-1e84-4ffa-aa57-eeb8ce7aceb8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:03.990897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1326/2664] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:03.995521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67828fe0-6c5f-47c7-864a-d658957f365c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.005350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67828fe0-6c5f-47c7-864a-d658957f365c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.009972 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debbaa3f-b499-4a6b-8eb2-5a26c3741c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.010560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbaa3f-b499-4a6b-8eb2-5a26c3741c78 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.010834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbaa3f-b499-4a6b-8eb2-5a26c3741c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.011103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbaa3f-b499-4a6b-8eb2-5a26c3741c78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.011768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbaa3f-b499-4a6b-8eb2-5a26c3741c78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.013613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbaa3f-b499-4a6b-8eb2-5a26c3741c78 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.024051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1339/2665] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:04.028197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1760f4e4-1e84-4ffa-aa57-eeb8ce7aceb8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.038098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1760f4e4-1e84-4ffa-aa57-eeb8ce7aceb8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.043428 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d9613-8d4d-440e-863e-2f2d32fda107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.044115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d9613-8d4d-440e-863e-2f2d32fda107 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.044397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d9613-8d4d-440e-863e-2f2d32fda107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.044666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d9613-8d4d-440e-863e-2f2d32fda107 admin admin] PATH_INFO: `/v3/projects/b8f41f4e03b540e8b057f5e13e98a4b3/users/a188d8483f814f3fbd67dc98bb0786d1/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.060170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d9613-8d4d-440e-863e-2f2d32fda107 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8f41f4e03b540e8b057f5e13e98a4b3, user_id=a188d8483f814f3fbd67dc98bb0786d1, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.062946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d9613-8d4d-440e-863e-2f2d32fda107 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.076676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1327/2666] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:04 2022] PUT /identity/v3/projects/b8f41f4e03b540e8b057f5e13e98a4b3/users/a188d8483f814f3fbd67dc98bb0786d1/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:04.086754 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202bd630-502d-483e-b738-3848a426df7a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.087037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202bd630-502d-483e-b738-3848a426df7a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.087310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202bd630-502d-483e-b738-3848a426df7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.115221 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-202bd630-502d-483e-b738-3848a426df7a None None] MFA Rules not processed for user `a188d8483f814f3fbd67dc98bb0786d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:04.116456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-202bd630-502d-483e-b738-3848a426df7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:04.179769 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1340/2667] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:04.184870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d9613-8d4d-440e-863e-2f2d32fda107 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.246237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d9613-8d4d-440e-863e-2f2d32fda107 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.250991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403c6c9e-170f-4d87-94ac-fe8108099856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.251585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403c6c9e-170f-4d87-94ac-fe8108099856 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.251890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403c6c9e-170f-4d87-94ac-fe8108099856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.252163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403c6c9e-170f-4d87-94ac-fe8108099856 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.252789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403c6c9e-170f-4d87-94ac-fe8108099856 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.254672 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403c6c9e-170f-4d87-94ac-fe8108099856 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.280465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1328/2668] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:21:04 2022] POST /identity/v3/projects => generated 385 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:04.284495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202bd630-502d-483e-b738-3848a426df7a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.295576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202bd630-502d-483e-b738-3848a426df7a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.342803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:04.353081 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cdc430-97b4-415b-857a-c85a66621431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.354277 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cdc430-97b4-415b-857a-c85a66621431 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.354573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cdc430-97b4-415b-857a-c85a66621431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.354831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cdc430-97b4-415b-857a-c85a66621431 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.355471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cdc430-97b4-415b-857a-c85a66621431 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.357456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cdc430-97b4-415b-857a-c85a66621431 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.377820 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1341/2669] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:21:04 2022] POST /identity/v3/users => generated 299 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:04.382020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403c6c9e-170f-4d87-94ac-fe8108099856 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.393182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403c6c9e-170f-4d87-94ac-fe8108099856 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.397791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f299af-e9bc-43b7-a08b-ca73a8d0927c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.398359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f299af-e9bc-43b7-a08b-ca73a8d0927c admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.398635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f299af-e9bc-43b7-a08b-ca73a8d0927c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.398906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f299af-e9bc-43b7-a08b-ca73a8d0927c admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.399541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f299af-e9bc-43b7-a08b-ca73a8d0927c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.401987 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f299af-e9bc-43b7-a08b-ca73a8d0927c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.412450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1329/2670] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:21:04.417247 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cdc430-97b4-415b-857a-c85a66621431 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.427136 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cdc430-97b4-415b-857a-c85a66621431 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.431727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31c8e47-0b8a-4ae3-9c0e-c96c056e7558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.432369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c8e47-0b8a-4ae3-9c0e-c96c056e7558 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.432643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c8e47-0b8a-4ae3-9c0e-c96c056e7558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.432913 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c8e47-0b8a-4ae3-9c0e-c96c056e7558 admin admin] PATH_INFO: `/v3/projects/3d7223db2949431d82d2d8f579a86e66/users/da6613dfbdab4b2ebf1c59bc1b64943d/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.448731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31c8e47-0b8a-4ae3-9c0e-c96c056e7558 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d7223db2949431d82d2d8f579a86e66, user_id=da6613dfbdab4b2ebf1c59bc1b64943d, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.451307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31c8e47-0b8a-4ae3-9c0e-c96c056e7558 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.464433 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1342/2671] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:04 2022] PUT /identity/v3/projects/3d7223db2949431d82d2d8f579a86e66/users/da6613dfbdab4b2ebf1c59bc1b64943d/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:04.468407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f299af-e9bc-43b7-a08b-ca73a8d0927c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.478294 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f299af-e9bc-43b7-a08b-ca73a8d0927c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.527305 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:04.535040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4c90ab-bc61-4157-ad12-fe9b25183a9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.535890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c90ab-bc61-4157-ad12-fe9b25183a9f admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.536415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c90ab-bc61-4157-ad12-fe9b25183a9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.536810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c90ab-bc61-4157-ad12-fe9b25183a9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.537646 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4c90ab-bc61-4157-ad12-fe9b25183a9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.539237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4c90ab-bc61-4157-ad12-fe9b25183a9f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.549449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1330/2672] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:04 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:04.553626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31c8e47-0b8a-4ae3-9c0e-c96c056e7558 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.563834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31c8e47-0b8a-4ae3-9c0e-c96c056e7558 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.568539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.569177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.569455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.569727 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] PATH_INFO: `/v3/projects/3d7223db2949431d82d2d8f579a86e66/users/da6613dfbdab4b2ebf1c59bc1b64943d/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.571901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d7223db2949431d82d2d8f579a86e66, user_id=da6613dfbdab4b2ebf1c59bc1b64943d, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.574526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.584526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1343/2673] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:04 2022] PUT /identity/v3/projects/3d7223db2949431d82d2d8f579a86e66/users/da6613dfbdab4b2ebf1c59bc1b64943d/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:04.594992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fbf135-980a-4241-b557-65faf283dfd5 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.595276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fbf135-980a-4241-b557-65faf283dfd5 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.595549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fbf135-980a-4241-b557-65faf283dfd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.623608 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-07fbf135-980a-4241-b557-65faf283dfd5 None None] MFA Rules not processed for user `da6613dfbdab4b2ebf1c59bc1b64943d`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:04.624848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-07fbf135-980a-4241-b557-65faf283dfd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:04.690781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1331/2674] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:04 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:04.702787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.704578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:04.761210 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05fa02f-5c97-445c-8154-df3d56103317 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.764881 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6002c21-5b1f-46f7-a476-9085305cceec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.765477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6002c21-5b1f-46f7-a476-9085305cceec service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.765759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6002c21-5b1f-46f7-a476-9085305cceec service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.766034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6002c21-5b1f-46f7-a476-9085305cceec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.775020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6002c21-5b1f-46f7-a476-9085305cceec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.776669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6002c21-5b1f-46f7-a476-9085305cceec service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.785922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1344/2675] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:04 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:04.815280 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fbf135-980a-4241-b557-65faf283dfd5 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:04.877585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fbf135-980a-4241-b557-65faf283dfd5 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:04.882242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:04.882862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.883140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.883413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] PATH_INFO: `/v3/users/a188d8483f814f3fbd67dc98bb0786d1` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.884559 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a188d8483f814f3fbd67dc98bb0786d1)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:04.886800 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:04.933020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a188d8483f814f3fbd67dc98bb0786d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:04.940239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a188d8483f814f3fbd67dc98bb0786d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:04.940898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a188d8483f814f3fbd67dc98bb0786d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:04.941521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a188d8483f814f3fbd67dc98bb0786d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:04.948905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a188d8483f814f3fbd67dc98bb0786d1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:04.959783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:04.967579 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1332/2676] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:21:04 2022] DELETE /identity/v3/users/a188d8483f814f3fbd67dc98bb0786d1 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:04.972190 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998b176-ef5f-4829-9338-b4fb7d52c240 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:04.972473 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998b176-ef5f-4829-9338-b4fb7d52c240 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:04.972745 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998b176-ef5f-4829-9338-b4fb7d52c240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:04.991538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7998b176-ef5f-4829-9338-b4fb7d52c240 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:04.992743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7998b176-ef5f-4829-9338-b4fb7d52c240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:05.050500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1345/2677] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:05.061164 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:05.062580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:05.123874 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725be252-d590-48ae-91e2-f133d80f581a admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:05.127516 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbb4f28-1764-4c3f-bdde-7ac3e56cd4a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:05.128162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbb4f28-1764-4c3f-bdde-7ac3e56cd4a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:05.128442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbb4f28-1764-4c3f-bdde-7ac3e56cd4a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:05.128716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbb4f28-1764-4c3f-bdde-7ac3e56cd4a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:05.137504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbb4f28-1764-4c3f-bdde-7ac3e56cd4a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:05.138929 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbb4f28-1764-4c3f-bdde-7ac3e56cd4a7 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:05.148431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1333/2678] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:05.408243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7998b176-ef5f-4829-9338-b4fb7d52c240 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:05.409667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7998b176-ef5f-4829-9338-b4fb7d52c240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:05.423129 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7998b176-ef5f-4829-9338-b4fb7d52c240 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:05.427891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:05.428526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:05.428803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:05.429148 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] PATH_INFO: `/v3/projects/b8f41f4e03b540e8b057f5e13e98a4b3` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:05.430309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8f41f4e03b540e8b057f5e13e98a4b3)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:05.432751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:05.475165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:05.482890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8f41f4e03b540e8b057f5e13e98a4b3', 'request_id': 'req-4ea52344-b4bf-4261-832b-86edf5d8fdad'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.490094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8f41f4e03b540e8b057f5e13e98a4b3', 'request_id': 'req-4ea52344-b4bf-4261-832b-86edf5d8fdad'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.495975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8f41f4e03b540e8b057f5e13e98a4b3', 'request_id': 'req-4ea52344-b4bf-4261-832b-86edf5d8fdad'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.507077 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] The token cache is being invalidate because project b8f41f4e03b540e8b057f5e13e98a4b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:21:05.507371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.508012 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.510400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1346/2679] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:21:05 2022] DELETE /identity/v3/projects/b8f41f4e03b540e8b057f5e13e98a4b3 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:05.514213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbb4f28-1764-4c3f-bdde-7ac3e56cd4a7 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:05.523767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbb4f28-1764-4c3f-bdde-7ac3e56cd4a7 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:05.582103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:05.582755 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:05.582985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:05.583253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] PATH_INFO: `/v3/users/da6613dfbdab4b2ebf1c59bc1b64943d` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:05.584375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] RBAC: Authorizing `identity:delete_user(user_id=da6613dfbdab4b2ebf1c59bc1b64943d)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:05.586273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:05.630527 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da6613dfbdab4b2ebf1c59bc1b64943d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.638146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da6613dfbdab4b2ebf1c59bc1b64943d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.638814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da6613dfbdab4b2ebf1c59bc1b64943d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.639409 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da6613dfbdab4b2ebf1c59bc1b64943d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.646246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-25571b5a-c357-4a9c-9641-f05b1c484301 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da6613dfbdab4b2ebf1c59bc1b64943d'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:05.663851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1334/2680] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:21:05 2022] DELETE /identity/v3/users/da6613dfbdab4b2ebf1c59bc1b64943d => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:05.928583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:05.930048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:05.944413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea52344-b4bf-4261-832b-86edf5d8fdad admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.002511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.003138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.003418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.003709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] PATH_INFO: `/v3/projects/3d7223db2949431d82d2d8f579a86e66` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.004819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d7223db2949431d82d2d8f579a86e66)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.006725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.054410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d7223db2949431d82d2d8f579a86e66', 'request_id': 'req-16baa5b5-8f1c-465e-90c9-c24f5b35546c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:06.062211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d7223db2949431d82d2d8f579a86e66', 'request_id': 'req-16baa5b5-8f1c-465e-90c9-c24f5b35546c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:06.068325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d7223db2949431d82d2d8f579a86e66', 'request_id': 'req-16baa5b5-8f1c-465e-90c9-c24f5b35546c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:06.080759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] The token cache is being invalidate because project 3d7223db2949431d82d2d8f579a86e66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:21:06.081082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:06.081733 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-16baa5b5-8f1c-465e-90c9-c24f5b35546c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:06.083223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1347/2681] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:21:05 2022] DELETE /identity/v3/projects/3d7223db2949431d82d2d8f579a86e66 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:06.088251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdfbdac-4ad0-48ee-b24f-6b8519457fd0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.088538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdfbdac-4ad0-48ee-b24f-6b8519457fd0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.088845 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdfbdac-4ad0-48ee-b24f-6b8519457fd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.109127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-dbdfbdac-4ad0-48ee-b24f-6b8519457fd0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:06.110590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-dbdfbdac-4ad0-48ee-b24f-6b8519457fd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:06.123913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:06.178019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1335/2682] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:06.192264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fdc02e-7067-4d61-aeec-f1ef2500d853 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.192623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fdc02e-7067-4d61-aeec-f1ef2500d853 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.192906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fdc02e-7067-4d61-aeec-f1ef2500d853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.213234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c0fdc02e-7067-4d61-aeec-f1ef2500d853 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:06.214439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c0fdc02e-7067-4d61-aeec-f1ef2500d853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:06.222486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1348/2683] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:06.226877 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdfbdac-4ad0-48ee-b24f-6b8519457fd0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.237021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdfbdac-4ad0-48ee-b24f-6b8519457fd0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.241544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.242140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.242440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.242711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.243357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:21:06.243652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.245505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.257876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1336/2684] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:21:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:21:06.262934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fdc02e-7067-4d61-aeec-f1ef2500d853 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.272515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fdc02e-7067-4d61-aeec-f1ef2500d853 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.277871 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb1b491-9fb8-4d1a-9cec-1038d9015a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.278412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb1b491-9fb8-4d1a-9cec-1038d9015a60 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.278692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb1b491-9fb8-4d1a-9cec-1038d9015a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.278966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb1b491-9fb8-4d1a-9cec-1038d9015a60 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.279613 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb1b491-9fb8-4d1a-9cec-1038d9015a60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.281594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb1b491-9fb8-4d1a-9cec-1038d9015a60 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.312804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1349/2685] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:21:06 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:06.318286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.331101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc8c5d2-7be8-47e1-89f2-91a1541bc6cf admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.390571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7a343e-58ea-43b5-8cb2-325d7a36ae56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.391192 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7a343e-58ea-43b5-8cb2-325d7a36ae56 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.391473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7a343e-58ea-43b5-8cb2-325d7a36ae56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.391842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7a343e-58ea-43b5-8cb2-325d7a36ae56 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.392492 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7a343e-58ea-43b5-8cb2-325d7a36ae56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.394497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7a343e-58ea-43b5-8cb2-325d7a36ae56 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.418336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1337/2686] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:21:06 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:06.422573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb1b491-9fb8-4d1a-9cec-1038d9015a60 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.433503 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb1b491-9fb8-4d1a-9cec-1038d9015a60 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.438578 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4751083-daf9-4768-a949-92983e885e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.439202 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4751083-daf9-4768-a949-92983e885e8f admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.439481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4751083-daf9-4768-a949-92983e885e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.439784 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4751083-daf9-4768-a949-92983e885e8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.440424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4751083-daf9-4768-a949-92983e885e8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.442112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4751083-daf9-4768-a949-92983e885e8f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.456214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1350/2687] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:06.457384 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7a343e-58ea-43b5-8cb2-325d7a36ae56 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.467972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7a343e-58ea-43b5-8cb2-325d7a36ae56 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.472596 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a59c76b-71a1-4a15-9b89-4d1145475d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.473201 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a59c76b-71a1-4a15-9b89-4d1145475d9e admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.473479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a59c76b-71a1-4a15-9b89-4d1145475d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.473753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a59c76b-71a1-4a15-9b89-4d1145475d9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.474396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a59c76b-71a1-4a15-9b89-4d1145475d9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.476026 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a59c76b-71a1-4a15-9b89-4d1145475d9e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.487858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1338/2688] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:06.492139 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4751083-daf9-4768-a949-92983e885e8f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.502419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4751083-daf9-4768-a949-92983e885e8f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.507044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa18912-4ad9-45da-a514-47de9232129c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.507705 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa18912-4ad9-45da-a514-47de9232129c admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.507998 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa18912-4ad9-45da-a514-47de9232129c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.508273 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa18912-4ad9-45da-a514-47de9232129c admin admin] PATH_INFO: `/v3/projects/df0255cdd58c443c8c39aa909b565cef/users/19badb17a5264b618b045b4f57e8d549/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.523105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa18912-4ad9-45da-a514-47de9232129c admin admin] RBAC: Authorizing `identity:create_grant(project_id=df0255cdd58c443c8c39aa909b565cef, user_id=19badb17a5264b618b045b4f57e8d549, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.526516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa18912-4ad9-45da-a514-47de9232129c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.529547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:06.541436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1351/2689] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:06 2022] PUT /identity/v3/projects/df0255cdd58c443c8c39aa909b565cef/users/19badb17a5264b618b045b4f57e8d549/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:06.551497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911ec055-7f80-4e30-910b-36339e6dcbe5 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.551866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911ec055-7f80-4e30-910b-36339e6dcbe5 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.552167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911ec055-7f80-4e30-910b-36339e6dcbe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.580240 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-911ec055-7f80-4e30-910b-36339e6dcbe5 None None] MFA Rules not processed for user `19badb17a5264b618b045b4f57e8d549`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:06.581445 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-911ec055-7f80-4e30-910b-36339e6dcbe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:06.645979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1339/2690] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:06.650712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa18912-4ad9-45da-a514-47de9232129c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.712207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa18912-4ad9-45da-a514-47de9232129c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.716891 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b060b8a4-c0c6-4355-b263-4a7c1d75d176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.717492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060b8a4-c0c6-4355-b263-4a7c1d75d176 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.717777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060b8a4-c0c6-4355-b263-4a7c1d75d176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.718055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060b8a4-c0c6-4355-b263-4a7c1d75d176 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.718677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b060b8a4-c0c6-4355-b263-4a7c1d75d176 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.720627 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b060b8a4-c0c6-4355-b263-4a7c1d75d176 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.747098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1352/2691] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:21:06 2022] POST /identity/v3/projects => generated 385 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:21:06.751212 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911ec055-7f80-4e30-910b-36339e6dcbe5 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.762362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911ec055-7f80-4e30-910b-36339e6dcbe5 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.770962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:06.819156 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1187a26-7bc3-4126-8a5f-f062fce29199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.819717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1187a26-7bc3-4126-8a5f-f062fce29199 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.820000 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1187a26-7bc3-4126-8a5f-f062fce29199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.820270 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1187a26-7bc3-4126-8a5f-f062fce29199 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.821148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1187a26-7bc3-4126-8a5f-f062fce29199 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.823337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1187a26-7bc3-4126-8a5f-f062fce29199 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.847499 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1340/2692] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:21:06 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:06.850848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b060b8a4-c0c6-4355-b263-4a7c1d75d176 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.862119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b060b8a4-c0c6-4355-b263-4a7c1d75d176 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.866758 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ceb5d5-88ab-475e-bbf2-bbabfe058c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.867355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ceb5d5-88ab-475e-bbf2-bbabfe058c88 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.867650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ceb5d5-88ab-475e-bbf2-bbabfe058c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.867947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ceb5d5-88ab-475e-bbf2-bbabfe058c88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.868592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ceb5d5-88ab-475e-bbf2-bbabfe058c88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.870263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ceb5d5-88ab-475e-bbf2-bbabfe058c88 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.881527 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1353/2693] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:21:06.885521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1187a26-7bc3-4126-8a5f-f062fce29199 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.895385 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1187a26-7bc3-4126-8a5f-f062fce29199 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.900027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552b5183-1af9-4a25-bc6a-6da20e7114f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.901207 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552b5183-1af9-4a25-bc6a-6da20e7114f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.901495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552b5183-1af9-4a25-bc6a-6da20e7114f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.901767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552b5183-1af9-4a25-bc6a-6da20e7114f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.902408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552b5183-1af9-4a25-bc6a-6da20e7114f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.904027 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552b5183-1af9-4a25-bc6a-6da20e7114f7 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.913830 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1341/2694] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:06.919162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ceb5d5-88ab-475e-bbf2-bbabfe058c88 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:06.929163 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ceb5d5-88ab-475e-bbf2-bbabfe058c88 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:06.934008 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db303baf-77c3-4434-93a3-b6c1c49baeb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:06.934668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db303baf-77c3-4434-93a3-b6c1c49baeb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.934935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db303baf-77c3-4434-93a3-b6c1c49baeb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.935211 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db303baf-77c3-4434-93a3-b6c1c49baeb6 admin admin] PATH_INFO: `/v3/projects/75789cbb0b7c4dfb83573d4b86472e13/users/7f4c182c24034c90b83476ea55c074df/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:06.950414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db303baf-77c3-4434-93a3-b6c1c49baeb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75789cbb0b7c4dfb83573d4b86472e13, user_id=7f4c182c24034c90b83476ea55c074df, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:06.953184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db303baf-77c3-4434-93a3-b6c1c49baeb6 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:06.966569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1354/2695] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:06 2022] PUT /identity/v3/projects/75789cbb0b7c4dfb83573d4b86472e13/users/7f4c182c24034c90b83476ea55c074df/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:06.976680 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c79212-69c7-4d83-90fe-e74a278fc998 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:06.976967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c79212-69c7-4d83-90fe-e74a278fc998 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:06.977274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c79212-69c7-4d83-90fe-e74a278fc998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.005808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-43c79212-69c7-4d83-90fe-e74a278fc998 None None] MFA Rules not processed for user `7f4c182c24034c90b83476ea55c074df`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:07.007009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-43c79212-69c7-4d83-90fe-e74a278fc998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:07.069984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1342/2696] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:07.074732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db303baf-77c3-4434-93a3-b6c1c49baeb6 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:07.138162 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db303baf-77c3-4434-93a3-b6c1c49baeb6 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:07.142794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c9f40-1501-457d-93ba-76849ae54001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:07.145694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c9f40-1501-457d-93ba-76849ae54001 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:07.145981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c9f40-1501-457d-93ba-76849ae54001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:07.148439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c9f40-1501-457d-93ba-76849ae54001 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.151261 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c9f40-1501-457d-93ba-76849ae54001 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:07.153191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c9f40-1501-457d-93ba-76849ae54001 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:07.179141 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1355/2697] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:21:07 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:21:07.184053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c79212-69c7-4d83-90fe-e74a278fc998 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:07.194992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c79212-69c7-4d83-90fe-e74a278fc998 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:07.252060 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bf0444-79ca-4e09-bbec-ad4c6cb2927f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:07.252662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf0444-79ca-4e09-bbec-ad4c6cb2927f admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:07.252944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf0444-79ca-4e09-bbec-ad4c6cb2927f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:07.253219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf0444-79ca-4e09-bbec-ad4c6cb2927f admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.253858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bf0444-79ca-4e09-bbec-ad4c6cb2927f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:07.255912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bf0444-79ca-4e09-bbec-ad4c6cb2927f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:07.277295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1343/2698] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:21:07 2022] POST /identity/v3/users => generated 299 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:07.281305 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c9f40-1501-457d-93ba-76849ae54001 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:07.291623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c9f40-1501-457d-93ba-76849ae54001 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:07.296791 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f65c05-b4df-4bbb-88ca-6cd36def3ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:07.297413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f65c05-b4df-4bbb-88ca-6cd36def3ee7 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:07.297699 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f65c05-b4df-4bbb-88ca-6cd36def3ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:07.297976 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f65c05-b4df-4bbb-88ca-6cd36def3ee7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.298618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f65c05-b4df-4bbb-88ca-6cd36def3ee7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:07.300250 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f65c05-b4df-4bbb-88ca-6cd36def3ee7 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:07.310912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1356/2699] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:07.315400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bf0444-79ca-4e09-bbec-ad4c6cb2927f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:07.326465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bf0444-79ca-4e09-bbec-ad4c6cb2927f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:07.330941 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ef0ef-7aea-4c5a-b866-afccbfd3f232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:07.331586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ef0ef-7aea-4c5a-b866-afccbfd3f232 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:07.331897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ef0ef-7aea-4c5a-b866-afccbfd3f232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:07.332176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ef0ef-7aea-4c5a-b866-afccbfd3f232 admin admin] PATH_INFO: `/v3/projects/5895ad08c10c4e2f8e73af4a3d6a39d8/users/9aff1d9bd200481780f0b6528de79550/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.342451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:07.350403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91ef0ef-7aea-4c5a-b866-afccbfd3f232 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5895ad08c10c4e2f8e73af4a3d6a39d8, user_id=9aff1d9bd200481780f0b6528de79550, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:07.353197 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91ef0ef-7aea-4c5a-b866-afccbfd3f232 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:07.365938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1344/2700] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:07 2022] PUT /identity/v3/projects/5895ad08c10c4e2f8e73af4a3d6a39d8/users/9aff1d9bd200481780f0b6528de79550/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:07.369843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f65c05-b4df-4bbb-88ca-6cd36def3ee7 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:07.379870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f65c05-b4df-4bbb-88ca-6cd36def3ee7 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:07.396502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:07.437945 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95217f06-3271-442c-b5a1-97ae1f5998e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:07.438549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95217f06-3271-442c-b5a1-97ae1f5998e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:07.438825 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95217f06-3271-442c-b5a1-97ae1f5998e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:07.439093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95217f06-3271-442c-b5a1-97ae1f5998e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.439748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95217f06-3271-442c-b5a1-97ae1f5998e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:07.441325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95217f06-3271-442c-b5a1-97ae1f5998e2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:07.451623 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1357/2701] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:07 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:07.455826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ef0ef-7aea-4c5a-b866-afccbfd3f232 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:07.466769 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ef0ef-7aea-4c5a-b866-afccbfd3f232 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:07.471326 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:07.472014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:07.472295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:07.472570 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] PATH_INFO: `/v3/projects/5895ad08c10c4e2f8e73af4a3d6a39d8/users/9aff1d9bd200481780f0b6528de79550/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.474470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5895ad08c10c4e2f8e73af4a3d6a39d8, user_id=9aff1d9bd200481780f0b6528de79550, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:07.477142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:07.487304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1345/2702] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:07 2022] PUT /identity/v3/projects/5895ad08c10c4e2f8e73af4a3d6a39d8/users/9aff1d9bd200481780f0b6528de79550/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:21:07.555127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b0dafb-4551-4bcb-84ab-0fd12d667d3a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:07.555411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b0dafb-4551-4bcb-84ab-0fd12d667d3a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:07.555709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b0dafb-4551-4bcb-84ab-0fd12d667d3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.585067 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-64b0dafb-4551-4bcb-84ab-0fd12d667d3a None None] MFA Rules not processed for user `9aff1d9bd200481780f0b6528de79550`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:07.586179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-64b0dafb-4551-4bcb-84ab-0fd12d667d3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:07.657004 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1358/2703] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:07 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:07.668371 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:07.670299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:07.727927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21301508-03fe-4671-83ec-3729f8156571 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:07.732420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acea5424-e104-4248-a887-7a7b50c27048 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:07.733019 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acea5424-e104-4248-a887-7a7b50c27048 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:07.733297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acea5424-e104-4248-a887-7a7b50c27048 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:07.733568 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acea5424-e104-4248-a887-7a7b50c27048 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:07.742444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acea5424-e104-4248-a887-7a7b50c27048 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:07.743886 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acea5424-e104-4248-a887-7a7b50c27048 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:07.753278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1346/2704] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:07 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:08.262166 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b0dafb-4551-4bcb-84ab-0fd12d667d3a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:08.263585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-64b0dafb-4551-4bcb-84ab-0fd12d667d3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:08.275323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b0dafb-4551-4bcb-84ab-0fd12d667d3a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:08.279184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:08.279810 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:08.280092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:08.280363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:08.344757 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:08.346349 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:08.356312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1359/2705] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:12.410685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acea5424-e104-4248-a887-7a7b50c27048 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:12.419325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acea5424-e104-4248-a887-7a7b50c27048 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:12.423185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49956bda-bc10-48cc-8399-11015f694668 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:12.423861 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49956bda-bc10-48cc-8399-11015f694668 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:12.424279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49956bda-bc10-48cc-8399-11015f694668 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:12.424551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49956bda-bc10-48cc-8399-11015f694668 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:12.482219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49956bda-bc10-48cc-8399-11015f694668 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:12.483636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49956bda-bc10-48cc-8399-11015f694668 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:12.492489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1347/2706] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:12 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:12.938062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:12.939630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:12.984511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=100 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:12.998911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c608521-06bc-42d3-85d9-4758d5bc8e50 service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:13.002502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ba3602-5f12-4ed3-b430-4b8a2b1d8f96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:13.003105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba3602-5f12-4ed3-b430-4b8a2b1d8f96 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:13.003381 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba3602-5f12-4ed3-b430-4b8a2b1d8f96 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:13.003651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba3602-5f12-4ed3-b430-4b8a2b1d8f96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:13.005351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-79ba3602-5f12-4ed3-b430-4b8a2b1d8f96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:13.074264 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ba3602-5f12-4ed3-b430-4b8a2b1d8f96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:13.075832 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ba3602-5f12-4ed3-b430-4b8a2b1d8f96 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:13.086171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1360/2707] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:14.617727 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49956bda-bc10-48cc-8399-11015f694668 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:14.629163 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49956bda-bc10-48cc-8399-11015f694668 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:14.633792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:14.634417 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:14.634696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:14.634971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] PATH_INFO: `/v3/users/19badb17a5264b618b045b4f57e8d549` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:14.636139 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=19badb17a5264b618b045b4f57e8d549)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:14.638799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:14.703890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19badb17a5264b618b045b4f57e8d549'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:14.712053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19badb17a5264b618b045b4f57e8d549'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:14.713314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19badb17a5264b618b045b4f57e8d549'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:14.714010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19badb17a5264b618b045b4f57e8d549'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:14.722565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19badb17a5264b618b045b4f57e8d549'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:14.728267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:14.743999 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1348/2708] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:21:14 2022] DELETE /identity/v3/users/19badb17a5264b618b045b4f57e8d549 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:14.749070 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc259526-9b22-4e16-b652-40bd9d358bf3 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:14.749591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc259526-9b22-4e16-b652-40bd9d358bf3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:14.749591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc259526-9b22-4e16-b652-40bd9d358bf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:14.769668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-bc259526-9b22-4e16-b652-40bd9d358bf3 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:14.771017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bc259526-9b22-4e16-b652-40bd9d358bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:14.838367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1361/2709] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:14.851538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:14.853168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:14.938108 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d342783a-9fcf-4cd4-bc32-5a9b65b32b0d admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:14.941890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5bccd8-07e5-4db7-ba7f-f734cf7b5bb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:14.942550 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5bccd8-07e5-4db7-ba7f-f734cf7b5bb2 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:14.942834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5bccd8-07e5-4db7-ba7f-f734cf7b5bb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:14.943107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5bccd8-07e5-4db7-ba7f-f734cf7b5bb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:14.955262 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5bccd8-07e5-4db7-ba7f-f734cf7b5bb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:14.956825 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5bccd8-07e5-4db7-ba7f-f734cf7b5bb2 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:14.968052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1349/2710] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:15.395539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc259526-9b22-4e16-b652-40bd9d358bf3 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:15.397058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bc259526-9b22-4e16-b652-40bd9d358bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:15.409698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc259526-9b22-4e16-b652-40bd9d358bf3 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:15.415411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:15.416189 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:15.416547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:15.416897 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] PATH_INFO: `/v3/projects/df0255cdd58c443c8c39aa909b565cef` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:15.418968 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df0255cdd58c443c8c39aa909b565cef)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:15.421337 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:15.472753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df0255cdd58c443c8c39aa909b565cef', 'request_id': 'req-ce4d591c-cf03-4dab-9048-538be28aa211'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.480990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df0255cdd58c443c8c39aa909b565cef', 'request_id': 'req-ce4d591c-cf03-4dab-9048-538be28aa211'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.487843 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df0255cdd58c443c8c39aa909b565cef', 'request_id': 'req-ce4d591c-cf03-4dab-9048-538be28aa211'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.499301 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] The token cache is being invalidate because project df0255cdd58c443c8c39aa909b565cef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:21:15.499591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.500263 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.501743 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1362/2711] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:21:15 2022] DELETE /identity/v3/projects/df0255cdd58c443c8c39aa909b565cef => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:15.505873 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5bccd8-07e5-4db7-ba7f-f734cf7b5bb2 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:15.518997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5bccd8-07e5-4db7-ba7f-f734cf7b5bb2 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:15.592761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:15.593411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:15.593692 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:15.593965 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] PATH_INFO: `/v3/users/7f4c182c24034c90b83476ea55c074df` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:15.595102 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f4c182c24034c90b83476ea55c074df)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:15.597080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:15.653993 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f4c182c24034c90b83476ea55c074df'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.661399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f4c182c24034c90b83476ea55c074df'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.662009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f4c182c24034c90b83476ea55c074df'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.662635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f4c182c24034c90b83476ea55c074df'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.669808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f4c182c24034c90b83476ea55c074df'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:15.688184 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1350/2712] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:21:15 2022] DELETE /identity/v3/users/7f4c182c24034c90b83476ea55c074df => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:15.977039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:15.979378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:16.006853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d591c-cf03-4dab-9048-538be28aa211 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:16.018614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:16.081593 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:16.082357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:16.082639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:16.082911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] PATH_INFO: `/v3/projects/75789cbb0b7c4dfb83573d4b86472e13` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:16.084529 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75789cbb0b7c4dfb83573d4b86472e13)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:16.086517 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:16.164585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75789cbb0b7c4dfb83573d4b86472e13', 'request_id': 'req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.176790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75789cbb0b7c4dfb83573d4b86472e13', 'request_id': 'req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.184091 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75789cbb0b7c4dfb83573d4b86472e13', 'request_id': 'req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.201140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] The token cache is being invalidate because project 75789cbb0b7c4dfb83573d4b86472e13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:21:16.201249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.201933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.203471 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1363/2713] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:21:15 2022] DELETE /identity/v3/projects/75789cbb0b7c4dfb83573d4b86472e13 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:16.207847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:16.209297 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:16.223655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe56c401-b52a-4c8f-8a39-c0ac3b16d3ba admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:16.228804 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:16.294074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:16.294871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:16.295252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:16.295626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] PATH_INFO: `/v3/users/9aff1d9bd200481780f0b6528de79550` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:16.296965 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9aff1d9bd200481780f0b6528de79550)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:16.304446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:16.383369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9aff1d9bd200481780f0b6528de79550'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.393634 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9aff1d9bd200481780f0b6528de79550'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.394126 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9aff1d9bd200481780f0b6528de79550'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.394848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9aff1d9bd200481780f0b6528de79550'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.403842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9aff1d9bd200481780f0b6528de79550'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.404464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:16.406535 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:16.430888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1351/2714] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:21:16 2022] DELETE /identity/v3/users/9aff1d9bd200481780f0b6528de79550 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:16.490014 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:16.491617 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:16.546984 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424104fd-fb2f-4e70-8cd1-185f9d71eaf1 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:16.584292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6300a1-a00e-4c99-b958-01edb2ef671e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:16.590569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:16.591203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:16.591701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:16.592429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] PATH_INFO: `/v3/users/00374d3d3bf649749781792273e35e3b` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:16.594270 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=00374d3d3bf649749781792273e35e3b)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:16.596235 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:16.610086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:16.644870 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:16.645761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:16.646106 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:16.646470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:16.648446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:16.658078 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:16.665323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:16.668329 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:16.679866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00374d3d3bf649749781792273e35e3b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.682667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1364/2715] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:21:16.690386 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00374d3d3bf649749781792273e35e3b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.691023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00374d3d3bf649749781792273e35e3b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.691891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00374d3d3bf649749781792273e35e3b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.700034 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00374d3d3bf649749781792273e35e3b'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:16.719502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1352/2716] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:21:16 2022] DELETE /identity/v3/users/00374d3d3bf649749781792273e35e3b => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:16.823220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:16.826237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:16.854917 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75f64d5-89a8-42ca-b232-797a974be1c0 service placement] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:16.934197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:16.934955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:16.935209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:16.935918 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] PATH_INFO: `/v3/projects/5895ad08c10c4e2f8e73af4a3d6a39d8` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:16.937386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] RBAC: Authorizing `identity:delete_project(project_id=5895ad08c10c4e2f8e73af4a3d6a39d8)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:16.939589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:16.947176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:16.948655 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:16.962959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899073ff-94ef-413c-827d-2a235dafbef1 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:16.970032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:16.971394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:16.971931 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:16.972394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] PATH_INFO: `/v3/projects/e76574d5f2094997a9c565d4ddf2a483` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:16.974888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=e76574d5f2094997a9c565d4ddf2a483)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:16.977482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.000767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5895ad08c10c4e2f8e73af4a3d6a39d8', 'request_id': 'req-b828b17e-0457-41aa-86f6-e1373c1a413e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.010555 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5895ad08c10c4e2f8e73af4a3d6a39d8', 'request_id': 'req-b828b17e-0457-41aa-86f6-e1373c1a413e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.017831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5895ad08c10c4e2f8e73af4a3d6a39d8', 'request_id': 'req-b828b17e-0457-41aa-86f6-e1373c1a413e'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.029880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] The token cache is being invalidate because project 5895ad08c10c4e2f8e73af4a3d6a39d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:21:17.030231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.030886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-b828b17e-0457-41aa-86f6-e1373c1a413e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.032090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e76574d5f2094997a9c565d4ddf2a483', 'request_id': 'req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.032472 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1365/2717] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:21:16 2022] DELETE /identity/v3/projects/5895ad08c10c4e2f8e73af4a3d6a39d8 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:17.039164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f2c62-03a8-4a05-95af-dd5558caa997 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.039948 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f2c62-03a8-4a05-95af-dd5558caa997 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.040230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f2c62-03a8-4a05-95af-dd5558caa997 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.046147 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e76574d5f2094997a9c565d4ddf2a483', 'request_id': 'req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.052259 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e76574d5f2094997a9c565d4ddf2a483', 'request_id': 'req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.062746 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-934f2c62-03a8-4a05-95af-dd5558caa997 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:17.064009 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-934f2c62-03a8-4a05-95af-dd5558caa997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:17.064224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] The token cache is being invalidate because project e76574d5f2094997a9c565d4ddf2a483 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:21:17.064616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.065682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e6bf4d69-b661-4d5e-b1f1-9d25208227ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:21:17.067799 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1353/2718] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:21:16 2022] DELETE /identity/v3/projects/e76574d5f2094997a9c565d4ddf2a483 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:17.074974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f939538-282a-4874-a965-a60034f2df52 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.075275 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f939538-282a-4874-a965-a60034f2df52 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.075544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f939538-282a-4874-a965-a60034f2df52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.095203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6f939538-282a-4874-a965-a60034f2df52 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:17.096436 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6f939538-282a-4874-a965-a60034f2df52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:17.132308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1366/2719] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:17.140878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4122327-494e-413a-9023-10a2ce797037 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.141170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4122327-494e-413a-9023-10a2ce797037 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.141446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4122327-494e-413a-9023-10a2ce797037 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.155108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:17.157218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1354/2720] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:17.162085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a4122327-494e-413a-9023-10a2ce797037 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:17.163884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a4122327-494e-413a-9023-10a2ce797037 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:17.165052 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927670ac-e35b-46d4-935b-ff66b614d873 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.165337 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927670ac-e35b-46d4-935b-ff66b614d873 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.165606 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927670ac-e35b-46d4-935b-ff66b614d873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.172781 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1367/2721] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:21:17.176903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4122327-494e-413a-9023-10a2ce797037 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.186608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4122327-494e-413a-9023-10a2ce797037 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.187292 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-927670ac-e35b-46d4-935b-ff66b614d873 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:17.188514 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-927670ac-e35b-46d4-935b-ff66b614d873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:17.191140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.191756 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.192041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.192314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.192954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:21:17.193255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.194851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.200453 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1355/2722] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:21:17.201340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927670ac-e35b-46d4-935b-ff66b614d873 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.210567 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1368/2723] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:21:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 15 14:21:17.210733 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927670ac-e35b-46d4-935b-ff66b614d873 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.211470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.215509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.216159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.216442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.216740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.217423 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:21:17.217732 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.219376 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.221391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe93cc-d639-475c-bc04-5a9b6d35cac5 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.226030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116c11fc-b0c8-42f0-8273-6a6093d2d2e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.226615 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116c11fc-b0c8-42f0-8273-6a6093d2d2e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.226898 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116c11fc-b0c8-42f0-8273-6a6093d2d2e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.227182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116c11fc-b0c8-42f0-8273-6a6093d2d2e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.227847 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116c11fc-b0c8-42f0-8273-6a6093d2d2e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.229747 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116c11fc-b0c8-42f0-8273-6a6093d2d2e3 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.231868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1356/2724] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:21:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:21:17.236183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.246653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c007278c-9de5-432f-a2ad-fec6e3136094 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.252222 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c71556-e7c3-4e75-bc4b-81eec367fd2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.252843 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c71556-e7c3-4e75-bc4b-81eec367fd2b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.253123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c71556-e7c3-4e75-bc4b-81eec367fd2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.253397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c71556-e7c3-4e75-bc4b-81eec367fd2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.254096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c71556-e7c3-4e75-bc4b-81eec367fd2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.256122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c71556-e7c3-4e75-bc4b-81eec367fd2b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.260876 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1369/2725] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:21:17.264924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116c11fc-b0c8-42f0-8273-6a6093d2d2e3 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.269740 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:17.291130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1357/2726] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:21:17.292141 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c71556-e7c3-4e75-bc4b-81eec367fd2b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.336109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116c11fc-b0c8-42f0-8273-6a6093d2d2e3 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.358082 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c71556-e7c3-4e75-bc4b-81eec367fd2b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.363181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98455928-a228-4cb0-b12d-a3e0aa67c962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.363817 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98455928-a228-4cb0-b12d-a3e0aa67c962 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.364503 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98455928-a228-4cb0-b12d-a3e0aa67c962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.364781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98455928-a228-4cb0-b12d-a3e0aa67c962 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.365415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98455928-a228-4cb0-b12d-a3e0aa67c962 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.367336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98455928-a228-4cb0-b12d-a3e0aa67c962 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.389564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1358/2727] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:17.393489 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98455928-a228-4cb0-b12d-a3e0aa67c962 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.397507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d95414-54ad-484a-8d7f-10a4d4fb2d30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.398119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d95414-54ad-484a-8d7f-10a4d4fb2d30 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.398569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d95414-54ad-484a-8d7f-10a4d4fb2d30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.398851 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d95414-54ad-484a-8d7f-10a4d4fb2d30 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.400240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d95414-54ad-484a-8d7f-10a4d4fb2d30 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.402246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d95414-54ad-484a-8d7f-10a4d4fb2d30 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.403394 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98455928-a228-4cb0-b12d-a3e0aa67c962 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.408032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedd067c-e198-46e3-aaa1-69f09323f8fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.408623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedd067c-e198-46e3-aaa1-69f09323f8fa admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.408903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedd067c-e198-46e3-aaa1-69f09323f8fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.409182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedd067c-e198-46e3-aaa1-69f09323f8fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.409823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedd067c-e198-46e3-aaa1-69f09323f8fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.411422 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedd067c-e198-46e3-aaa1-69f09323f8fa admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.422300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1359/2728] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:21:17.424023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1370/2729] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/users => generated 287 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:21:17.426397 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedd067c-e198-46e3-aaa1-69f09323f8fa admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.427892 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d95414-54ad-484a-8d7f-10a4d4fb2d30 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.437320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedd067c-e198-46e3-aaa1-69f09323f8fa admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.438396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d95414-54ad-484a-8d7f-10a4d4fb2d30 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.442613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5e33f1-a93e-4e10-a734-2476f5de41f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.443109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8cb3a4-4c0d-439b-8d49-396a41b14f4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.443227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e33f1-a93e-4e10-a734-2476f5de41f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.443461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e33f1-a93e-4e10-a734-2476f5de41f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.443704 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8cb3a4-4c0d-439b-8d49-396a41b14f4a admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.443824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5e33f1-a93e-4e10-a734-2476f5de41f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.444020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8cb3a4-4c0d-439b-8d49-396a41b14f4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.444280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8cb3a4-4c0d-439b-8d49-396a41b14f4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.444391 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5e33f1-a93e-4e10-a734-2476f5de41f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.444940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8cb3a4-4c0d-439b-8d49-396a41b14f4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.446302 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5e33f1-a93e-4e10-a734-2476f5de41f8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.446817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8cb3a4-4c0d-439b-8d49-396a41b14f4a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.457065 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1371/2730] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:21:17.457228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1360/2731] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:21:17.460827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8cb3a4-4c0d-439b-8d49-396a41b14f4a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.461048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5e33f1-a93e-4e10-a734-2476f5de41f8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.471159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8cb3a4-4c0d-439b-8d49-396a41b14f4a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.471307 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5e33f1-a93e-4e10-a734-2476f5de41f8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.475724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e90abd9-0ae3-4994-a82a-3b684283ccc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.475876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6739530-93e2-45e9-b4e4-c808ad2782a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.476323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e90abd9-0ae3-4994-a82a-3b684283ccc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.476500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6739530-93e2-45e9-b4e4-c808ad2782a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.476601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e90abd9-0ae3-4994-a82a-3b684283ccc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.476784 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6739530-93e2-45e9-b4e4-c808ad2782a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.476893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e90abd9-0ae3-4994-a82a-3b684283ccc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.477066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6739530-93e2-45e9-b4e4-c808ad2782a8 admin admin] PATH_INFO: `/v3/projects/d1c7aba744814c7ca12a1dab68e1815a/users/37aa36608fc94226be02046ed4bbaf35/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.477483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e90abd9-0ae3-4994-a82a-3b684283ccc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.479071 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e90abd9-0ae3-4994-a82a-3b684283ccc4 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.491144 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1372/2732] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:17.492807 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6739530-93e2-45e9-b4e4-c808ad2782a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1c7aba744814c7ca12a1dab68e1815a, user_id=37aa36608fc94226be02046ed4bbaf35, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.495241 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e90abd9-0ae3-4994-a82a-3b684283ccc4 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.495461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6739530-93e2-45e9-b4e4-c808ad2782a8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.505658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e90abd9-0ae3-4994-a82a-3b684283ccc4 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.508670 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1361/2733] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:17 2022] PUT /identity/v3/projects/d1c7aba744814c7ca12a1dab68e1815a/users/37aa36608fc94226be02046ed4bbaf35/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:21:17.513666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d574009-08fa-4c14-9ff8-67bd87deddb5 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.513950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d574009-08fa-4c14-9ff8-67bd87deddb5 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.514248 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d574009-08fa-4c14-9ff8-67bd87deddb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.533959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-3d574009-08fa-4c14-9ff8-67bd87deddb5 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:17.535168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3d574009-08fa-4c14-9ff8-67bd87deddb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:17.544812 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:17.565516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3156d7d-746b-4e2d-9c42-8de63ec079e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.566177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3156d7d-746b-4e2d-9c42-8de63ec079e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.566455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3156d7d-746b-4e2d-9c42-8de63ec079e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.566730 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3156d7d-746b-4e2d-9c42-8de63ec079e1 admin admin] PATH_INFO: `/v3/projects/3280b84c87ee422c986300291186a41d/users/177058e8e42244faabca6c572a581428/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.581675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3156d7d-746b-4e2d-9c42-8de63ec079e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3280b84c87ee422c986300291186a41d, user_id=177058e8e42244faabca6c572a581428, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.584380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3156d7d-746b-4e2d-9c42-8de63ec079e1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.599072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1373/2734] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:17 2022] PUT /identity/v3/projects/3280b84c87ee422c986300291186a41d/users/177058e8e42244faabca6c572a581428/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:21:17.601923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1362/2735] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:21:17.604787 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418a674-1d1e-47ac-a150-cf43666ee83c None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.606112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418a674-1d1e-47ac-a150-cf43666ee83c None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.606112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418a674-1d1e-47ac-a150-cf43666ee83c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.612446 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d574009-08fa-4c14-9ff8-67bd87deddb5 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.613882 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3d574009-08fa-4c14-9ff8-67bd87deddb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:17.624493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7418a674-1d1e-47ac-a150-cf43666ee83c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:17.625665 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7418a674-1d1e-47ac-a150-cf43666ee83c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:17.648841 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:17.672061 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d574009-08fa-4c14-9ff8-67bd87deddb5 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.685894 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1374/2736] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:17.695970 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418a674-1d1e-47ac-a150-cf43666ee83c None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:17.720759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5867dd24-fa12-420c-a992-c1a6293d94cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.721368 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5867dd24-fa12-420c-a992-c1a6293d94cb service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.721644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5867dd24-fa12-420c-a992-c1a6293d94cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.721916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5867dd24-fa12-420c-a992-c1a6293d94cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.731157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5867dd24-fa12-420c-a992-c1a6293d94cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.732620 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5867dd24-fa12-420c-a992-c1a6293d94cb service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.742058 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1363/2737] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:21:17.750341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418a674-1d1e-47ac-a150-cf43666ee83c None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:17.753956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0e4f88-ccbf-4c8f-a29e-9cff4603bd27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:17.754562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e4f88-ccbf-4c8f-a29e-9cff4603bd27 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:17.754842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e4f88-ccbf-4c8f-a29e-9cff4603bd27 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:17.755114 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e4f88-ccbf-4c8f-a29e-9cff4603bd27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:17.764944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0e4f88-ccbf-4c8f-a29e-9cff4603bd27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:17.766668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0e4f88-ccbf-4c8f-a29e-9cff4603bd27 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:17.777606 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1375/2738] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:21:23.239389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01281c2-2081-4046-8391-45c22b563c79 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.240020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01281c2-2081-4046-8391-45c22b563c79 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.240020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01281c2-2081-4046-8391-45c22b563c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.287948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e01281c2-2081-4046-8391-45c22b563c79 None None] MFA Rules not processed for user `177058e8e42244faabca6c572a581428`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:23.289193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e01281c2-2081-4046-8391-45c22b563c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:23.353257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff867566-585c-4d95-abcf-343fc9b1f2a1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.353867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff867566-585c-4d95-abcf-343fc9b1f2a1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.353867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff867566-585c-4d95-abcf-343fc9b1f2a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.398514 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-ff867566-585c-4d95-abcf-343fc9b1f2a1 None None] MFA Rules not processed for user `37aa36608fc94226be02046ed4bbaf35`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:23.400014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ff867566-585c-4d95-abcf-343fc9b1f2a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:23.422551 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1364/2739] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:23.427555 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01281c2-2081-4046-8391-45c22b563c79 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:23.448387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01281c2-2081-4046-8391-45c22b563c79 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:23.455021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2dcb9d-f1ec-470a-85bf-be2d382521c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:23.455786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2dcb9d-f1ec-470a-85bf-be2d382521c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.456136 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2dcb9d-f1ec-470a-85bf-be2d382521c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.456482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2dcb9d-f1ec-470a-85bf-be2d382521c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.457198 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2dcb9d-f1ec-470a-85bf-be2d382521c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:23.459186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2dcb9d-f1ec-470a-85bf-be2d382521c7 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:23.498315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:23.500319 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1365/2740] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:21:23 2022] POST /identity/v3/projects => generated 371 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:23.501505 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2dcb9d-f1ec-470a-85bf-be2d382521c7 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:23.525429 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1376/2741] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:21:23.533420 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff867566-585c-4d95-abcf-343fc9b1f2a1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:23.534483 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ff867566-585c-4d95-abcf-343fc9b1f2a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:23.602863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2dcb9d-f1ec-470a-85bf-be2d382521c7 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:23.608952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00d87b2-6543-4711-944b-37cc968d7af0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:23.610090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00d87b2-6543-4711-944b-37cc968d7af0 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.611072 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00d87b2-6543-4711-944b-37cc968d7af0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.611575 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00d87b2-6543-4711-944b-37cc968d7af0 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.612475 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00d87b2-6543-4711-944b-37cc968d7af0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:23.614918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00d87b2-6543-4711-944b-37cc968d7af0 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:23.624862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:23.648770 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff867566-585c-4d95-abcf-343fc9b1f2a1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:23.650350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1366/2742] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:21:23 2022] POST /identity/v3/users => generated 292 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:21:23.653480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00d87b2-6543-4711-944b-37cc968d7af0 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:23.670496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00d87b2-6543-4711-944b-37cc968d7af0 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:23.676404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c5aba0-7b47-4e8d-97b0-d1bca26fd271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:23.677211 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c5aba0-7b47-4e8d-97b0-d1bca26fd271 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.677533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c5aba0-7b47-4e8d-97b0-d1bca26fd271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.677903 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c5aba0-7b47-4e8d-97b0-d1bca26fd271 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.678957 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c5aba0-7b47-4e8d-97b0-d1bca26fd271 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:23.681057 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c5aba0-7b47-4e8d-97b0-d1bca26fd271 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:23.695752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1367/2743] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:23 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:23.698193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c5aba0-7b47-4e8d-97b0-d1bca26fd271 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:23.713984 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c5aba0-7b47-4e8d-97b0-d1bca26fd271 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:23.720625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bd68dd-15fa-4010-98d9-434a6fab0daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:23.721431 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd68dd-15fa-4010-98d9-434a6fab0daa admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.721824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd68dd-15fa-4010-98d9-434a6fab0daa admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.722209 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd68dd-15fa-4010-98d9-434a6fab0daa admin admin] PATH_INFO: `/v3/projects/021d14bb7c7343419dfde7ab08d8b384/users/72df4aad4a9b44afbb23456c0e955d47/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.732713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f135392-dd99-48be-b781-eecfe40ada38 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:23.733353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f135392-dd99-48be-b781-eecfe40ada38 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.733631 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f135392-dd99-48be-b781-eecfe40ada38 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.733912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f135392-dd99-48be-b781-eecfe40ada38 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.746071 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bd68dd-15fa-4010-98d9-434a6fab0daa admin admin] RBAC: Authorizing `identity:create_grant(project_id=021d14bb7c7343419dfde7ab08d8b384, user_id=72df4aad4a9b44afbb23456c0e955d47, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:23.750317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bd68dd-15fa-4010-98d9-434a6fab0daa admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:23.779540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1368/2744] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:23 2022] PUT /identity/v3/projects/021d14bb7c7343419dfde7ab08d8b384/users/72df4aad4a9b44afbb23456c0e955d47/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:21:23.783348 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bd68dd-15fa-4010-98d9-434a6fab0daa admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:23.818621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f135392-dd99-48be-b781-eecfe40ada38 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:23.820776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f135392-dd99-48be-b781-eecfe40ada38 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:23.833491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1377/2745] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:21:23.867040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bd68dd-15fa-4010-98d9-434a6fab0daa admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:23.872404 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8a20d2-d71f-496e-bffb-30360123a83e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:23.873206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8a20d2-d71f-496e-bffb-30360123a83e admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.873678 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8a20d2-d71f-496e-bffb-30360123a83e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.874115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8a20d2-d71f-496e-bffb-30360123a83e admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.874945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8a20d2-d71f-496e-bffb-30360123a83e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:23.877043 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8a20d2-d71f-496e-bffb-30360123a83e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:23.892218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1369/2746] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:21:23 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:21:23.896717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f135392-dd99-48be-b781-eecfe40ada38 service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:23.922807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f135392-dd99-48be-b781-eecfe40ada38 service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:23.931582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:23.932522 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:23.933589 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:23.934496 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] PATH_INFO: `/v3/projects/021d14bb7c7343419dfde7ab08d8b384/users/72df4aad4a9b44afbb23456c0e955d47/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:23.936666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] RBAC: Authorizing `identity:create_grant(project_id=021d14bb7c7343419dfde7ab08d8b384, user_id=72df4aad4a9b44afbb23456c0e955d47, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:23.939722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:23.952415 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1378/2747] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:21:23 2022] PUT /identity/v3/projects/021d14bb7c7343419dfde7ab08d8b384/users/72df4aad4a9b44afbb23456c0e955d47/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:21:25.687644 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0001b7d-5588-41ad-a023-ddedba221870 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:25.688739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0001b7d-5588-41ad-a023-ddedba221870 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:25.689177 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0001b7d-5588-41ad-a023-ddedba221870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:25.713083 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c0001b7d-5588-41ad-a023-ddedba221870 None None] MFA Rules not processed for user `37aa36608fc94226be02046ed4bbaf35`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:25.715001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c0001b7d-5588-41ad-a023-ddedba221870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:25.802095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1370/2748] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:26.594967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:26.597314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:26.628785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:26.724307 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664cf631-0724-4d90-b0fd-43efe1ce3958 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:26.729974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:26.730955 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:26.731455 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:26.731982 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:26.734422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:26.748327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:26.750278 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:26.764421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1379/2749] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:30.408321 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9e03b8-ea88-4f31-86a1-c4dbcf47eb08 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:30.409318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9e03b8-ea88-4f31-86a1-c4dbcf47eb08 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:30.409751 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9e03b8-ea88-4f31-86a1-c4dbcf47eb08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:30.412238 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:30.420245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:30.443668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-bd9e03b8-ea88-4f31-86a1-c4dbcf47eb08 None None] MFA Rules not processed for user `72df4aad4a9b44afbb23456c0e955d47`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:30.445746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bd9e03b8-ea88-4f31-86a1-c4dbcf47eb08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:30.506507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:30.523423 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb74cfa2-21c7-4654-aaa6-25467fd143fd service placement] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:30.529725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8f85c2-045d-400c-9e42-828f5bc29097 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:30.530353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8f85c2-045d-400c-9e42-828f5bc29097 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:30.530628 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8f85c2-045d-400c-9e42-828f5bc29097 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:30.530901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8f85c2-045d-400c-9e42-828f5bc29097 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:30.533565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6a8f85c2-045d-400c-9e42-828f5bc29097 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:30.545410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8f85c2-045d-400c-9e42-828f5bc29097 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:30.546853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8f85c2-045d-400c-9e42-828f5bc29097 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:30.550101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1371/2750] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:30 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:21:30.566662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9e03b8-ea88-4f31-86a1-c4dbcf47eb08 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:30.574544 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1380/2751] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:30 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:21:30.670965 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9e03b8-ea88-4f31-86a1-c4dbcf47eb08 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:30.677481 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a051510-c168-4c3f-b9f6-f0bbe5a4dc1f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:30.678109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a051510-c168-4c3f-b9f6-f0bbe5a4dc1f service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:30.678396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a051510-c168-4c3f-b9f6-f0bbe5a4dc1f service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:30.678665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a051510-c168-4c3f-b9f6-f0bbe5a4dc1f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:30.757924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a051510-c168-4c3f-b9f6-f0bbe5a4dc1f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:30.759500 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a051510-c168-4c3f-b9f6-f0bbe5a4dc1f service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:30.769204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1372/2752] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:32.278908 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e3cbc-51ee-47e6-bd51-29a2ce316330 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:32.279430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e3cbc-51ee-47e6-bd51-29a2ce316330 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:32.279430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e3cbc-51ee-47e6-bd51-29a2ce316330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:32.298158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-f49e3cbc-51ee-47e6-bd51-29a2ce316330 None None] MFA Rules not processed for user `177058e8e42244faabca6c572a581428`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:21:32.299341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f49e3cbc-51ee-47e6-bd51-29a2ce316330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:32.306176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1381/2753] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:21:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:21:38.121087 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a051510-c168-4c3f-b9f6-f0bbe5a4dc1f service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:38.136168 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a051510-c168-4c3f-b9f6-f0bbe5a4dc1f service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:38.146144 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:38.147167 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:38.147447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:38.147770 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:38.149567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:38.168835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:38.171933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:38.196011 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1373/2754] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:21:42.292786 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=53 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:48.179700 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=71 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:21:53.155650 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49e3cbc-51ee-47e6-bd51-29a2ce316330 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:21:53.165174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49e3cbc-51ee-47e6-bd51-29a2ce316330 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:21:53.168748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:21:53.169347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:21:53.169624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:21:53.169896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:21:53.171723 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:21:53.181037 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:21:53.182481 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:21:53.191234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1382/2755] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:21:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:22:03.187635 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:22:10.307133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:22:10.317485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876857e-26d0-4b56-a534-e5312a0f8d0c service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:22:10.321327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2707b4c8-5d06-4c5a-bbdf-fc7a168c2318 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:22:10.321926 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2707b4c8-5d06-4c5a-bbdf-fc7a168c2318 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:22:10.322206 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2707b4c8-5d06-4c5a-bbdf-fc7a168c2318 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:22:10.322482 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2707b4c8-5d06-4c5a-bbdf-fc7a168c2318 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:22:10.332020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2707b4c8-5d06-4c5a-bbdf-fc7a168c2318 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:22:10.333432 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2707b4c8-5d06-4c5a-bbdf-fc7a168c2318 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:22:10.342731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1374/2756] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:22:10 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:22:11.891616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:22:11.893499 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:22:11.909675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10887054-d13b-461e-983e-993780b7d3b3 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:22:11.915095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:22:11.915712 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:22:11.916002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:22:11.916270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:22:11.917969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:22:11.930347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:22:11.931921 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:22:11.942944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1383/2757] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:22:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:22:20.338544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:22:21.938597 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:23:07.124127 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02373307-4b0a-4128-8af1-40a0a45f28a8 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:23:07.124675 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02373307-4b0a-4128-8af1-40a0a45f28a8 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:23:07.124796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02373307-4b0a-4128-8af1-40a0a45f28a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:23:07.143059 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-02373307-4b0a-4128-8af1-40a0a45f28a8 None None] MFA Rules not processed for user `37aa36608fc94226be02046ed4bbaf35`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:23:07.144242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-02373307-4b0a-4128-8af1-40a0a45f28a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:23:07.150857 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1375/2758] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:23:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:23:07.459924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:23:07.469553 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f666a749-bb13-4038-8992-d9a6929905be service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:23:07.473315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:23:07.473915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:23:07.474197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:23:07.474470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:23:07.476302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:23:07.486834 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:23:07.488286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:23:07.496249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1384/2759] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:23:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:23:17.138276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:23:17.492654 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:24:38.584598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02373307-4b0a-4128-8af1-40a0a45f28a8 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:24:38.595623 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02373307-4b0a-4128-8af1-40a0a45f28a8 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:24:38.599363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:24:38.599995 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:24:38.600282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:24:38.600558 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:24:38.602312 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:24:38.617392 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:24:38.618812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:24:38.630595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1376/2760] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:24:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:24:39.755675 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:24:39.765480 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792c6ce2-ea55-44d7-a228-10b03af60cb2 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:24:39.769565 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:24:39.770208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:24:39.770484 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:24:39.770755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:24:39.772562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:24:39.787610 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:24:39.789083 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:24:39.801226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1385/2761] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:24:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:24:40.136279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:24:40.137870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:24:40.152725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a63a3-620a-4891-907c-5efd6e85c9f3 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:24:40.158070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:24:40.158679 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:24:40.158959 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:24:40.159255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:24:40.161129 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:24:40.173735 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:24:40.175293 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:24:40.186522 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1377/2762] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:24:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:24:41.799135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:24:41.809259 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0b2c4a-06a4-4319-9a31-7621c5100f19 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:24:41.814304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711e17f0-ed0d-4e65-b8b7-57317fe450e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:24:41.815179 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711e17f0-ed0d-4e65-b8b7-57317fe450e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:24:41.815474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711e17f0-ed0d-4e65-b8b7-57317fe450e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:24:41.816058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711e17f0-ed0d-4e65-b8b7-57317fe450e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:24:41.818424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-711e17f0-ed0d-4e65-b8b7-57317fe450e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:24:41.829958 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711e17f0-ed0d-4e65-b8b7-57317fe450e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:24:41.831528 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711e17f0-ed0d-4e65-b8b7-57317fe450e9 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:24:41.845502 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1386/2763] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:24:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:24:42.969103 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:24:42.978274 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679cecb-60c2-4577-a232-36169414edd6 service placement] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:24:42.981852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:24:42.982439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:24:42.982712 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:24:42.982976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:24:42.985115 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:24:42.999032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:24:43.001395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:24:43.019981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1378/2764] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:24:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:24:50.263248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2582a61d-ffe4-4edf-b35c-d3ca9bcd391a None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:24:50.263775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2582a61d-ffe4-4edf-b35c-d3ca9bcd391a None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:24:50.263866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2582a61d-ffe4-4edf-b35c-d3ca9bcd391a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:24:50.283739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2582a61d-ffe4-4edf-b35c-d3ca9bcd391a None None] MFA Rules not processed for user `37aa36608fc94226be02046ed4bbaf35`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:24:50.285254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2582a61d-ffe4-4edf-b35c-d3ca9bcd391a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:24:50.293096 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1387/2765] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:24:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:24:53.008170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=18 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:25:00.277904 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=15 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:26:23.659510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:23.661037 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:23.721966 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1c8944-ed90-4528-9a86-cfa1c811a949 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:23.725488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c1481a-610b-47f1-8b4a-96e98af43ae5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:23.726083 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c1481a-610b-47f1-8b4a-96e98af43ae5 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:23.726357 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c1481a-610b-47f1-8b4a-96e98af43ae5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:23.726625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c1481a-610b-47f1-8b4a-96e98af43ae5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:23.734668 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c1481a-610b-47f1-8b4a-96e98af43ae5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:23.736718 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c1481a-610b-47f1-8b4a-96e98af43ae5 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:23.745186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1379/2766] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:26:33.086662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2582a61d-ffe4-4edf-b35c-d3ca9bcd391a None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:33.111942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2582a61d-ffe4-4edf-b35c-d3ca9bcd391a None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:33.118421 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:33.119032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:33.119498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:33.119867 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:33.121668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:33.136357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:33.137965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:33.163495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1388/2767] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:26:33.741298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=26 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:26:36.463506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa4150-071a-410c-8cbd-f0e1308449db None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:36.464183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa4150-071a-410c-8cbd-f0e1308449db None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:36.464183 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa4150-071a-410c-8cbd-f0e1308449db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:36.484582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-3baa4150-071a-410c-8cbd-f0e1308449db None None] MFA Rules not processed for user `37aa36608fc94226be02046ed4bbaf35`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:26:36.485869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-3baa4150-071a-410c-8cbd-f0e1308449db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:36.493846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1380/2768] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:26:36 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:26:37.105058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:37.117516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cee541-47c2-4243-9c9f-8512327e3e37 service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:37.122790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d32724d-6e00-4286-9450-9eb8bb79b243 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:37.123398 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32724d-6e00-4286-9450-9eb8bb79b243 service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:37.123700 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32724d-6e00-4286-9450-9eb8bb79b243 service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:37.123991 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32724d-6e00-4286-9450-9eb8bb79b243 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:37.134088 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d32724d-6e00-4286-9450-9eb8bb79b243 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:37.135708 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d32724d-6e00-4286-9450-9eb8bb79b243 service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:37.146255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1389/2769] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:26:39.010650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa4150-071a-410c-8cbd-f0e1308449db None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:39.021967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa4150-071a-410c-8cbd-f0e1308449db None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:39.026087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d32724d-6e00-4286-9450-9eb8bb79b243 service placement] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:39.026573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383aeb82-5231-4373-aa89-f0f710c8ab61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:39.026919 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383aeb82-5231-4373-aa89-f0f710c8ab61 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:39.027354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383aeb82-5231-4373-aa89-f0f710c8ab61 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:39.027641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383aeb82-5231-4373-aa89-f0f710c8ab61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:39.037237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383aeb82-5231-4373-aa89-f0f710c8ab61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:39.037409 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d32724d-6e00-4286-9450-9eb8bb79b243 service placement] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:39.039507 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383aeb82-5231-4373-aa89-f0f710c8ab61 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:39.041247 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a0d79-c0c7-497d-b27c-529b27797fa6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:39.041840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a0d79-c0c7-497d-b27c-529b27797fa6 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:39.042119 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a0d79-c0c7-497d-b27c-529b27797fa6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:39.042391 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a0d79-c0c7-497d-b27c-529b27797fa6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:39.050903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a0d79-c0c7-497d-b27c-529b27797fa6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:39.051334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1381/2770] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:39 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:26:39.052389 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a0d79-c0c7-497d-b27c-529b27797fa6 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:39.062378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1390/2771] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:26:45.773077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383aeb82-5231-4373-aa89-f0f710c8ab61 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:45.785967 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383aeb82-5231-4373-aa89-f0f710c8ab61 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:45.791414 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334c8d10-a8ce-4643-9cb7-d293ee4bf550 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:45.792049 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c8d10-a8ce-4643-9cb7-d293ee4bf550 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:45.792328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c8d10-a8ce-4643-9cb7-d293ee4bf550 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:45.792598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c8d10-a8ce-4643-9cb7-d293ee4bf550 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:45.802253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334c8d10-a8ce-4643-9cb7-d293ee4bf550 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:45.803932 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334c8d10-a8ce-4643-9cb7-d293ee4bf550 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:45.815234 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1382/2772] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:26:46.313348 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a0d79-c0c7-497d-b27c-529b27797fa6 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:46.323438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a0d79-c0c7-497d-b27c-529b27797fa6 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:46.327220 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bbff0c-6464-4925-b429-05dca0e43b19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:46.327859 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bbff0c-6464-4925-b429-05dca0e43b19 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:46.328150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bbff0c-6464-4925-b429-05dca0e43b19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:46.328426 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bbff0c-6464-4925-b429-05dca0e43b19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:46.388956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bbff0c-6464-4925-b429-05dca0e43b19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:46.390406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bbff0c-6464-4925-b429-05dca0e43b19 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:46.400869 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1391/2773] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 15 14:26:49.687997 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334c8d10-a8ce-4643-9cb7-d293ee4bf550 service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:49.706864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334c8d10-a8ce-4643-9cb7-d293ee4bf550 service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:49.727324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:49.728338 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:49.728801 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:49.729433 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:49.731439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:49.753953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:49.756662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:49.796304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1383/2774] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:26:53.396424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bbff0c-6464-4925-b429-05dca0e43b19 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:53.406376 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bbff0c-6464-4925-b429-05dca0e43b19 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:53.411842 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:53.412477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:53.412755 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:53.413028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] PATH_INFO: `/v3/users/177058e8e42244faabca6c572a581428` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:53.414157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] RBAC: Authorizing `identity:delete_user(user_id=177058e8e42244faabca6c572a581428)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:53.416113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:53.463154 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '177058e8e42244faabca6c572a581428'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.471033 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '177058e8e42244faabca6c572a581428'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.471732 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '177058e8e42244faabca6c572a581428'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.472343 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '177058e8e42244faabca6c572a581428'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.479310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '177058e8e42244faabca6c572a581428'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.496803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1392/2775] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:26:53 2022] DELETE /identity/v3/users/177058e8e42244faabca6c572a581428 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:26:53.670721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:53.683400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64567ce-c626-4b44-895e-85c0cdfbeb33 service placement] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:53.741298 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:53.741930 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:53.742213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:53.742488 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] PATH_INFO: `/v3/projects/3280b84c87ee422c986300291186a41d` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:53.743593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] RBAC: Authorizing `identity:delete_project(project_id=3280b84c87ee422c986300291186a41d)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:53.745540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:53.791317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3280b84c87ee422c986300291186a41d', 'request_id': 'req-156f36a4-bcb9-4a89-81a1-501a18223aad'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.798904 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3280b84c87ee422c986300291186a41d', 'request_id': 'req-156f36a4-bcb9-4a89-81a1-501a18223aad'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.804530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3280b84c87ee422c986300291186a41d', 'request_id': 'req-156f36a4-bcb9-4a89-81a1-501a18223aad'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.816244 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] The token cache is being invalidate because project 3280b84c87ee422c986300291186a41d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:26:53.816533 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.817162 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.818660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1384/2776] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:26:53 2022] DELETE /identity/v3/projects/3280b84c87ee422c986300291186a41d => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:26:53.822981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:53.824413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:53.837815 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293dfea-201e-4f2f-881a-dec861308491 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:53.862174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:26:53.895934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:53.896558 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:53.896838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:53.897116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] PATH_INFO: `/v3/users/72df4aad4a9b44afbb23456c0e955d47` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:53.898243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=72df4aad4a9b44afbb23456c0e955d47)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:53.900205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:53.945902 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72df4aad4a9b44afbb23456c0e955d47'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.953333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72df4aad4a9b44afbb23456c0e955d47'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.954041 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72df4aad4a9b44afbb23456c0e955d47'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.954682 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72df4aad4a9b44afbb23456c0e955d47'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.962311 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-cd79a407-ec5d-4706-a659-39d0dde3dd2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72df4aad4a9b44afbb23456c0e955d47'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:53.980506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1393/2777] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:26:53 2022] DELETE /identity/v3/users/72df4aad4a9b44afbb23456c0e955d47 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:26:54.150759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:54.152208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:54.166822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156f36a4-bcb9-4a89-81a1-501a18223aad admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:54.211779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:26:54.228098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:54.228730 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.229012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.229291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] PATH_INFO: `/v3/projects/021d14bb7c7343419dfde7ab08d8b384` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.230400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=021d14bb7c7343419dfde7ab08d8b384)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:54.232421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:54.277818 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '021d14bb7c7343419dfde7ab08d8b384', 'request_id': 'req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:54.285991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '021d14bb7c7343419dfde7ab08d8b384', 'request_id': 'req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:54.291540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '021d14bb7c7343419dfde7ab08d8b384', 'request_id': 'req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:54.303247 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] The token cache is being invalidate because project 021d14bb7c7343419dfde7ab08d8b384 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:26:54.303541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:54.304215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ae9dcb3-8367-4b0a-9320-a8b6a24a40f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:26:54.305702 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1385/2778] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:26:54 2022] DELETE /identity/v3/projects/021d14bb7c7343419dfde7ab08d8b384 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:26:54.310734 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd36e80-d58d-4b23-a8dc-67358def0952 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.311023 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd36e80-d58d-4b23-a8dc-67358def0952 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.311325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd36e80-d58d-4b23-a8dc-67358def0952 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.331896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-edd36e80-d58d-4b23-a8dc-67358def0952 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:26:54.333093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-edd36e80-d58d-4b23-a8dc-67358def0952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:54.391598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1394/2779] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:26:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:26:54.399725 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ccfdf1-7794-4c31-9d84-eaee342ba73b None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.399969 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ccfdf1-7794-4c31-9d84-eaee342ba73b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.400246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ccfdf1-7794-4c31-9d84-eaee342ba73b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.419868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-00ccfdf1-7794-4c31-9d84-eaee342ba73b None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:26:54.421098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-00ccfdf1-7794-4c31-9d84-eaee342ba73b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:54.429753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1386/2780] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:26:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:26:54.436103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd36e80-d58d-4b23-a8dc-67358def0952 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:54.446710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd36e80-d58d-4b23-a8dc-67358def0952 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:54.451306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:54.451933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.452216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.452489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.453137 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:26:54.453438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:54.455072 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:54.466355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1395/2781] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:26:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:26:54.470821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ccfdf1-7794-4c31-9d84-eaee342ba73b None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:54.480020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ccfdf1-7794-4c31-9d84-eaee342ba73b None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:54.484637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c98a8a-2128-448c-8bfb-adb557eebcb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:54.485225 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c98a8a-2128-448c-8bfb-adb557eebcb3 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.485502 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c98a8a-2128-448c-8bfb-adb557eebcb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.486377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c98a8a-2128-448c-8bfb-adb557eebcb3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.487304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c98a8a-2128-448c-8bfb-adb557eebcb3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:54.489313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c98a8a-2128-448c-8bfb-adb557eebcb3 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:54.516002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1387/2782] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:26:54 2022] POST /identity/v3/projects => generated 379 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:26:54.520773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:54.530698 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248dfa6d-d211-4b2a-a3be-961e1aadb01c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:54.560896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:26:54.589456 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1163c6ed-d00c-4a5a-97ae-7455ac328bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:54.590054 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1163c6ed-d00c-4a5a-97ae-7455ac328bcc admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.590330 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1163c6ed-d00c-4a5a-97ae-7455ac328bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.590600 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1163c6ed-d00c-4a5a-97ae-7455ac328bcc admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.591216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1163c6ed-d00c-4a5a-97ae-7455ac328bcc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:54.593169 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1163c6ed-d00c-4a5a-97ae-7455ac328bcc admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:54.614411 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1396/2783] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:26:54 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:26:54.618460 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c98a8a-2128-448c-8bfb-adb557eebcb3 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:54.629299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c98a8a-2128-448c-8bfb-adb557eebcb3 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:54.633893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069ebd90-db4e-4e96-8b18-149c91e6fb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:54.634480 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069ebd90-db4e-4e96-8b18-149c91e6fb53 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.634757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069ebd90-db4e-4e96-8b18-149c91e6fb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.635029 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069ebd90-db4e-4e96-8b18-149c91e6fb53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.635665 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069ebd90-db4e-4e96-8b18-149c91e6fb53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:54.637271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069ebd90-db4e-4e96-8b18-149c91e6fb53 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:54.647549 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1388/2784] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:26:54.651493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1163c6ed-d00c-4a5a-97ae-7455ac328bcc admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:54.661568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1163c6ed-d00c-4a5a-97ae-7455ac328bcc admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:54.666117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3058c9db-cfb6-4c81-a5c2-30cac590d739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:54.666701 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058c9db-cfb6-4c81-a5c2-30cac590d739 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.666977 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058c9db-cfb6-4c81-a5c2-30cac590d739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.667249 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058c9db-cfb6-4c81-a5c2-30cac590d739 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.667909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058c9db-cfb6-4c81-a5c2-30cac590d739 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:54.669507 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058c9db-cfb6-4c81-a5c2-30cac590d739 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:54.681232 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1397/2785] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:26:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:26:54.685324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069ebd90-db4e-4e96-8b18-149c91e6fb53 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:54.696254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069ebd90-db4e-4e96-8b18-149c91e6fb53 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:54.700905 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:54.701544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.701824 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.702098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] PATH_INFO: `/v3/projects/b9f4deb43acf410aa0a8e88c20ddb3f5/users/1c072306d6534925b92fbc5ef01aaaf6/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.717190 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9f4deb43acf410aa0a8e88c20ddb3f5, user_id=1c072306d6534925b92fbc5ef01aaaf6, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:54.719875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:54.732979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1389/2786] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:26:54 2022] PUT /identity/v3/projects/b9f4deb43acf410aa0a8e88c20ddb3f5/users/1c072306d6534925b92fbc5ef01aaaf6/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:26:54.740344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a0108-c7eb-4cde-b9e2-6108b3808b24 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.740629 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a0108-c7eb-4cde-b9e2-6108b3808b24 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.740901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a0108-c7eb-4cde-b9e2-6108b3808b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.760387 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-f39a0108-c7eb-4cde-b9e2-6108b3808b24 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:26:54.761580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f39a0108-c7eb-4cde-b9e2-6108b3808b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:54.820833 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1398/2787] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:26:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:26:54.831001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:54.832744 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:54.889581 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb03d314-b54a-4aaf-9a4f-d68f47577448 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:54.894130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b17a2-4e7d-44d4-b857-1c6e76a6f287 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:54.894728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b17a2-4e7d-44d4-b857-1c6e76a6f287 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:54.895010 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b17a2-4e7d-44d4-b857-1c6e76a6f287 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:54.895286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b17a2-4e7d-44d4-b857-1c6e76a6f287 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:54.904229 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148b17a2-4e7d-44d4-b857-1c6e76a6f287 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:54.905580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148b17a2-4e7d-44d4-b857-1c6e76a6f287 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:54.915932 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1390/2788] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:26:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:26:58.767462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f6390-d314-4bab-9a3b-95061c0e6873 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:58.767979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f6390-d314-4bab-9a3b-95061c0e6873 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:58.768197 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775f6390-d314-4bab-9a3b-95061c0e6873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:58.797556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-775f6390-d314-4bab-9a3b-95061c0e6873 None None] MFA Rules not processed for user `1c072306d6534925b92fbc5ef01aaaf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:26:58.798776 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-775f6390-d314-4bab-9a3b-95061c0e6873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:26:58.872201 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1399/2789] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:26:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:26:58.876708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b17a2-4e7d-44d4-b857-1c6e76a6f287 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:58.887088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b17a2-4e7d-44d4-b857-1c6e76a6f287 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:58.891842 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf0822c-5b74-4016-824b-6fa95c0f0f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:58.892541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf0822c-5b74-4016-824b-6fa95c0f0f71 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:58.892897 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf0822c-5b74-4016-824b-6fa95c0f0f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:58.893181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf0822c-5b74-4016-824b-6fa95c0f0f71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:58.893862 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf0822c-5b74-4016-824b-6fa95c0f0f71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:58.895970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf0822c-5b74-4016-824b-6fa95c0f0f71 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:58.907093 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:26:58.930067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1391/2790] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:26:58 2022] POST /identity/v3/projects => generated 377 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:26:58.931206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775f6390-d314-4bab-9a3b-95061c0e6873 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:58.943093 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775f6390-d314-4bab-9a3b-95061c0e6873 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:58.984257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:26:59.001862 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3721ea-604a-4cd5-a5f9-d6aa52982ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:59.002294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3721ea-604a-4cd5-a5f9-d6aa52982ced admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:59.002655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3721ea-604a-4cd5-a5f9-d6aa52982ced admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:59.003515 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3721ea-604a-4cd5-a5f9-d6aa52982ced admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:59.004442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3721ea-604a-4cd5-a5f9-d6aa52982ced admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:59.006372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3721ea-604a-4cd5-a5f9-d6aa52982ced admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:59.033298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1400/2791] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:26:58 2022] POST /identity/v3/users => generated 295 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:26:59.037399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf0822c-5b74-4016-824b-6fa95c0f0f71 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:59.050885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf0822c-5b74-4016-824b-6fa95c0f0f71 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:59.057358 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d67cad-83b3-40d0-b616-60fc8ca56c6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:59.057980 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d67cad-83b3-40d0-b616-60fc8ca56c6c admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:59.058257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d67cad-83b3-40d0-b616-60fc8ca56c6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:59.058526 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d67cad-83b3-40d0-b616-60fc8ca56c6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:59.059178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d67cad-83b3-40d0-b616-60fc8ca56c6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:59.060810 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d67cad-83b3-40d0-b616-60fc8ca56c6c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:59.075983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1392/2792] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:26:59 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:26:59.077109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3721ea-604a-4cd5-a5f9-d6aa52982ced admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:59.089308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3721ea-604a-4cd5-a5f9-d6aa52982ced admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:59.094437 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db683e0c-4bdc-430a-a319-794be81711e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:59.095359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db683e0c-4bdc-430a-a319-794be81711e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:59.095645 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db683e0c-4bdc-430a-a319-794be81711e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:59.095950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db683e0c-4bdc-430a-a319-794be81711e9 admin admin] PATH_INFO: `/v3/projects/16e1410c94744608a40b482c96aca5f7/users/0268010872084a15bc890ba2bc431e02/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:59.113294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db683e0c-4bdc-430a-a319-794be81711e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16e1410c94744608a40b482c96aca5f7, user_id=0268010872084a15bc890ba2bc431e02, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:59.115975 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db683e0c-4bdc-430a-a319-794be81711e9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:59.132742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1401/2793] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:26:59 2022] PUT /identity/v3/projects/16e1410c94744608a40b482c96aca5f7/users/0268010872084a15bc890ba2bc431e02/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:26:59.136756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d67cad-83b3-40d0-b616-60fc8ca56c6c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:59.149030 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d67cad-83b3-40d0-b616-60fc8ca56c6c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:59.248520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bb722c-064b-4fc3-b0d9-25c32707af9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:59.249131 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bb722c-064b-4fc3-b0d9-25c32707af9d admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:59.249408 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bb722c-064b-4fc3-b0d9-25c32707af9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:59.249674 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bb722c-064b-4fc3-b0d9-25c32707af9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:59.250311 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bb722c-064b-4fc3-b0d9-25c32707af9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:59.251944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bb722c-064b-4fc3-b0d9-25c32707af9d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:59.263111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1393/2794] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:26:59 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:26:59.267298 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db683e0c-4bdc-430a-a319-794be81711e9 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:26:59.282403 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db683e0c-4bdc-430a-a319-794be81711e9 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:26:59.287155 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:26:59.287887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:26:59.288183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:26:59.288459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] PATH_INFO: `/v3/projects/16e1410c94744608a40b482c96aca5f7/users/0268010872084a15bc890ba2bc431e02/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:26:59.290412 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16e1410c94744608a40b482c96aca5f7, user_id=0268010872084a15bc890ba2bc431e02, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:26:59.293187 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:26:59.304710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1402/2795] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:26:59 2022] PUT /identity/v3/projects/16e1410c94744608a40b482c96aca5f7/users/0268010872084a15bc890ba2bc431e02/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:27:03.808213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e314300-d50d-4232-97e3-7f93d3e6430a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:27:03.808760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e314300-d50d-4232-97e3-7f93d3e6430a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:27:03.808760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e314300-d50d-4232-97e3-7f93d3e6430a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:27:03.837485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-0e314300-d50d-4232-97e3-7f93d3e6430a None None] MFA Rules not processed for user `0268010872084a15bc890ba2bc431e02`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:27:03.838597 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-0e314300-d50d-4232-97e3-7f93d3e6430a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:27:03.909759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1394/2796] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:27:03 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:27:03.928146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:27:03.929895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:27:04.007399 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf72c52-69e3-46cc-af45-17ceb49a8220 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:27:04.012930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab619c6-ad87-497d-b71b-73962d5088f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:27:04.013541 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab619c6-ad87-497d-b71b-73962d5088f4 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:27:04.013823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab619c6-ad87-497d-b71b-73962d5088f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:27:04.014098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab619c6-ad87-497d-b71b-73962d5088f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:27:04.074192 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab619c6-ad87-497d-b71b-73962d5088f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:27:04.075944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab619c6-ad87-497d-b71b-73962d5088f4 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:27:04.087890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1403/2797] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:27:03 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:27:13.905716 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=71,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:27:14.083560 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=81,keystone:INSERT=5 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:27:25.649438 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ed237-2ed8-4b8e-9282-5d53d1e91faf None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:27:25.649954 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ed237-2ed8-4b8e-9282-5d53d1e91faf None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:27:25.650065 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ed237-2ed8-4b8e-9282-5d53d1e91faf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:27:25.668791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b07ed237-2ed8-4b8e-9282-5d53d1e91faf None None] MFA Rules not processed for user `1c072306d6534925b92fbc5ef01aaaf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:27:25.669972 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b07ed237-2ed8-4b8e-9282-5d53d1e91faf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:27:25.677483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1395/2798] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:27:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:27:25.989919 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab619c6-ad87-497d-b71b-73962d5088f4 service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:27:26.043601 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab619c6-ad87-497d-b71b-73962d5088f4 service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:27:26.047244 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:27:26.047883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:27:26.048170 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:27:26.048448 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:27:26.050206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:27:26.060321 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:27:26.061739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:27:26.069808 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1404/2799] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:27:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:27:29.079135 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07ed237-2ed8-4b8e-9282-5d53d1e91faf None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:27:29.080876 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b07ed237-2ed8-4b8e-9282-5d53d1e91faf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:27:29.098594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07ed237-2ed8-4b8e-9282-5d53d1e91faf None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:27:29.104694 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:27:29.105413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:27:29.105789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:27:29.106137 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:27:29.108050 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:27:29.121217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:27:29.122942 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:27:29.135130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1396/2800] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:27:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:27:36.066049 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=26 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:27:39.131290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=9 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:27:43.227737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:27:43.236290 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab102dc-fd10-4d40-a2a3-65a4b3047d71 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:27:43.240531 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:27:43.241208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:27:43.241490 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:27:43.241765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:27:43.250505 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:27:43.251963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:27:43.261427 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1405/2801] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:27:43 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:27:53.257292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:24.396688 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:24.432902 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1501c82c-18c2-4093-a439-bfae13c9adcc service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:24.436509 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:24.437101 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:24.437381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:24.437654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:24.439363 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:24.551566 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:24.553576 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:24.573040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1397/2802] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:27.443027 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:27.444934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:27.486598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d006e8-c691-41f1-9549-7b2f8d0042e1 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:27.492068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:27.492695 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:27.492974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:27.493246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] PATH_INFO: `/v3/users/37aa36608fc94226be02046ed4bbaf35` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:27.494362 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=37aa36608fc94226be02046ed4bbaf35)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:27.496315 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:27.542573 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37aa36608fc94226be02046ed4bbaf35'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.549719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37aa36608fc94226be02046ed4bbaf35'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.550352 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37aa36608fc94226be02046ed4bbaf35'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.551021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37aa36608fc94226be02046ed4bbaf35'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.558598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-e490d7de-cc49-48dd-b41e-1a146fa011a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37aa36608fc94226be02046ed4bbaf35'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.576274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1406/2803] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:27 2022] DELETE /identity/v3/users/37aa36608fc94226be02046ed4bbaf35 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:27.742236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:27.753375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b76f38-5e45-4e2a-981c-8d8c7229785f service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:27.811662 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:27.812314 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:27.812598 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:27.812875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] PATH_INFO: `/v3/projects/d1c7aba744814c7ca12a1dab68e1815a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:27.813994 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1c7aba744814c7ca12a1dab68e1815a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:27.815924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:27.827191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=100 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:27.861270 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1c7aba744814c7ca12a1dab68e1815a', 'request_id': 'req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.867622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1c7aba744814c7ca12a1dab68e1815a', 'request_id': 'req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.873032 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1c7aba744814c7ca12a1dab68e1815a', 'request_id': 'req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.883963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] The token cache is being invalidate because project d1c7aba744814c7ca12a1dab68e1815a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:27.884252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.884881 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-a6d677bb-1e82-4e81-91ad-3a51ea6ffb2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:27.886320 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1398/2804] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:27 2022] DELETE /identity/v3/projects/d1c7aba744814c7ca12a1dab68e1815a => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:27.891186 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db1279-1c0e-4d64-9251-0545c8500420 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:27.891474 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db1279-1c0e-4d64-9251-0545c8500420 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:27.891785 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db1279-1c0e-4d64-9251-0545c8500420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:27.950979 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-b5db1279-1c0e-4d64-9251-0545c8500420 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:27.952138 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-b5db1279-1c0e-4d64-9251-0545c8500420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:28.010629 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1407/2805] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:28.018504 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a1f01-eee4-4b70-9de8-9d284b563aed None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.018859 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a1f01-eee4-4b70-9de8-9d284b563aed None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.019228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a1f01-eee4-4b70-9de8-9d284b563aed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.038711 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c76a1f01-eee4-4b70-9de8-9d284b563aed None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:28.039934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c76a1f01-eee4-4b70-9de8-9d284b563aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:28.047604 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1399/2806] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:28.051737 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5db1279-1c0e-4d64-9251-0545c8500420 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:28.062062 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5db1279-1c0e-4d64-9251-0545c8500420 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:28.066681 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:28.067309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.067588 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.067889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.069283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:28:28.069648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:28.071495 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:28.083048 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1408/2807] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:28:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:28:28.087245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76a1f01-eee4-4b70-9de8-9d284b563aed None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:28.096219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76a1f01-eee4-4b70-9de8-9d284b563aed None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:28.101883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449e171c-31db-43ba-9ef8-93fd2fc1c157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:28.102478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e171c-31db-43ba-9ef8-93fd2fc1c157 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.102753 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e171c-31db-43ba-9ef8-93fd2fc1c157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.103021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e171c-31db-43ba-9ef8-93fd2fc1c157 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.103653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449e171c-31db-43ba-9ef8-93fd2fc1c157 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:28.105564 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449e171c-31db-43ba-9ef8-93fd2fc1c157 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:28.133666 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1400/2808] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:28 2022] POST /identity/v3/projects => generated 405 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:28:28.138002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:28.144256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:28.148425 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1f0a9c-9020-47bc-ac83-0763b801656e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:28.205802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b653e-3c2f-42d9-b79e-511ac9c83cd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:28.206402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b653e-3c2f-42d9-b79e-511ac9c83cd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.206683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b653e-3c2f-42d9-b79e-511ac9c83cd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.206957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b653e-3c2f-42d9-b79e-511ac9c83cd8 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.207591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4b653e-3c2f-42d9-b79e-511ac9c83cd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:28.209543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4b653e-3c2f-42d9-b79e-511ac9c83cd8 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:28.230774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1409/2809] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:28 2022] POST /identity/v3/users => generated 303 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:28.235080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449e171c-31db-43ba-9ef8-93fd2fc1c157 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:28.245834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449e171c-31db-43ba-9ef8-93fd2fc1c157 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:28.250407 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4e79fa-0bec-41e3-aff7-08b058a9edc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:28.251002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e79fa-0bec-41e3-aff7-08b058a9edc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.251278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e79fa-0bec-41e3-aff7-08b058a9edc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.251552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e79fa-0bec-41e3-aff7-08b058a9edc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.252242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4e79fa-0bec-41e3-aff7-08b058a9edc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:28.253808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4e79fa-0bec-41e3-aff7-08b058a9edc8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:28.263998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1401/2810] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:28.267759 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b653e-3c2f-42d9-b79e-511ac9c83cd8 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:28.277889 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b653e-3c2f-42d9-b79e-511ac9c83cd8 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:28.282520 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd69c27b-a5d6-4436-93a9-651aa9080516 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:28.283107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd69c27b-a5d6-4436-93a9-651aa9080516 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.283386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd69c27b-a5d6-4436-93a9-651aa9080516 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.283658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd69c27b-a5d6-4436-93a9-651aa9080516 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.284327 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd69c27b-a5d6-4436-93a9-651aa9080516 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:28.285911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd69c27b-a5d6-4436-93a9-651aa9080516 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:28.296817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1410/2811] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:28.300911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4e79fa-0bec-41e3-aff7-08b058a9edc8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:28.311850 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4e79fa-0bec-41e3-aff7-08b058a9edc8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:28.316425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:28.317064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.317336 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.317610 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] PATH_INFO: `/v3/projects/8ffaa93476c94ff9beabdacf24864a4d/users/c50f032f8bc54f0d89cbd308a7bba511/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.333073 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ffaa93476c94ff9beabdacf24864a4d, user_id=c50f032f8bc54f0d89cbd308a7bba511, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:28.335787 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:28.348166 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1402/2812] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:28 2022] PUT /identity/v3/projects/8ffaa93476c94ff9beabdacf24864a4d/users/c50f032f8bc54f0d89cbd308a7bba511/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:28.354279 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4ff9b5-0a1d-45e4-8e26-a2a93489441f None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.354861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4ff9b5-0a1d-45e4-8e26-a2a93489441f None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.355265 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4ff9b5-0a1d-45e4-8e26-a2a93489441f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.374686 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-0a4ff9b5-0a1d-45e4-8e26-a2a93489441f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:28.376001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-0a4ff9b5-0a1d-45e4-8e26-a2a93489441f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:28.433709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1411/2813] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:28:28.445405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:28.447356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:28.518413 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d8b82c-7c5e-4685-97e2-ed61b4e46d78 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:28.522213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59451a73-e964-424d-9d57-0f37f895a21c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:28.522815 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59451a73-e964-424d-9d57-0f37f895a21c service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:28.523095 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59451a73-e964-424d-9d57-0f37f895a21c service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:28.523369 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59451a73-e964-424d-9d57-0f37f895a21c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:28.532269 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59451a73-e964-424d-9d57-0f37f895a21c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:28.533683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59451a73-e964-424d-9d57-0f37f895a21c service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:28.544231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1403/2814] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:29.176778 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235fed1-9c8d-4a8a-93f6-b1a6aedc96fa None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:29.177333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235fed1-9c8d-4a8a-93f6-b1a6aedc96fa None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:29.177333 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235fed1-9c8d-4a8a-93f6-b1a6aedc96fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:29.206218 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-4235fed1-9c8d-4a8a-93f6-b1a6aedc96fa None None] MFA Rules not processed for user `c50f032f8bc54f0d89cbd308a7bba511`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:29.207419 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4235fed1-9c8d-4a8a-93f6-b1a6aedc96fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:29.278799 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1412/2815] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:29 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:29.283440 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59451a73-e964-424d-9d57-0f37f895a21c service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:29.293449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59451a73-e964-424d-9d57-0f37f895a21c service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:29.298023 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa45f53c-5315-4169-a251-a15f8452736e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:29.298626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa45f53c-5315-4169-a251-a15f8452736e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:29.299090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa45f53c-5315-4169-a251-a15f8452736e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:29.299511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa45f53c-5315-4169-a251-a15f8452736e admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:29.300328 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa45f53c-5315-4169-a251-a15f8452736e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:29.302396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa45f53c-5315-4169-a251-a15f8452736e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:29.335742 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1404/2816] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:29 2022] POST /identity/v3/projects => generated 405 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:29.338231 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4235fed1-9c8d-4a8a-93f6-b1a6aedc96fa None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:29.348971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4235fed1-9c8d-4a8a-93f6-b1a6aedc96fa None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:29.358378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:29.406966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6d5063-155a-4951-9e21-19fcc73bba2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:29.407580 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6d5063-155a-4951-9e21-19fcc73bba2c admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:29.407895 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6d5063-155a-4951-9e21-19fcc73bba2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:29.408175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6d5063-155a-4951-9e21-19fcc73bba2c admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:29.408817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6d5063-155a-4951-9e21-19fcc73bba2c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:29.410742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6d5063-155a-4951-9e21-19fcc73bba2c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:29.432590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1413/2817] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:29 2022] POST /identity/v3/users => generated 309 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:29.436544 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa45f53c-5315-4169-a251-a15f8452736e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:29.447538 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa45f53c-5315-4169-a251-a15f8452736e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:29.452118 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dcbfd8-3521-4e82-8f95-797b1c11d8b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:29.452708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dcbfd8-3521-4e82-8f95-797b1c11d8b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:29.452986 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dcbfd8-3521-4e82-8f95-797b1c11d8b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:29.453257 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dcbfd8-3521-4e82-8f95-797b1c11d8b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:29.453899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dcbfd8-3521-4e82-8f95-797b1c11d8b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:29.455473 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dcbfd8-3521-4e82-8f95-797b1c11d8b6 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:29.456781 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:29.467322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1405/2818] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:29.471274 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6d5063-155a-4951-9e21-19fcc73bba2c admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:29.481304 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6d5063-155a-4951-9e21-19fcc73bba2c admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:29.485903 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2323cfc2-ab0a-4881-a1b7-9bebd8732597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:29.486547 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2323cfc2-ab0a-4881-a1b7-9bebd8732597 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:29.486829 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2323cfc2-ab0a-4881-a1b7-9bebd8732597 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:29.487108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2323cfc2-ab0a-4881-a1b7-9bebd8732597 admin admin] PATH_INFO: `/v3/projects/bebf021bb9764059b6d1ef95ca0ca966/users/32320c57852b40df84dafeb97e98bf1c/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:29.502568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2323cfc2-ab0a-4881-a1b7-9bebd8732597 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bebf021bb9764059b6d1ef95ca0ca966, user_id=32320c57852b40df84dafeb97e98bf1c, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:29.505344 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2323cfc2-ab0a-4881-a1b7-9bebd8732597 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:29.518454 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1414/2819] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:29 2022] PUT /identity/v3/projects/bebf021bb9764059b6d1ef95ca0ca966/users/32320c57852b40df84dafeb97e98bf1c/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:29.522946 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dcbfd8-3521-4e82-8f95-797b1c11d8b6 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:29.532630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dcbfd8-3521-4e82-8f95-797b1c11d8b6 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:29.588552 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdc4acf-ad0c-459d-b073-f7d913c6cc93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:29.589080 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdc4acf-ad0c-459d-b073-f7d913c6cc93 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:29.589364 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdc4acf-ad0c-459d-b073-f7d913c6cc93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:29.589641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdc4acf-ad0c-459d-b073-f7d913c6cc93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:29.590300 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdc4acf-ad0c-459d-b073-f7d913c6cc93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:29.591976 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdc4acf-ad0c-459d-b073-f7d913c6cc93 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:29.602323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1406/2820] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:29 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:29.606772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2323cfc2-ab0a-4881-a1b7-9bebd8732597 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:29.617034 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2323cfc2-ab0a-4881-a1b7-9bebd8732597 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:29.621632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:29.622312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:29.622594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:29.622872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] PATH_INFO: `/v3/projects/bebf021bb9764059b6d1ef95ca0ca966/users/32320c57852b40df84dafeb97e98bf1c/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:29.624860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bebf021bb9764059b6d1ef95ca0ca966, user_id=32320c57852b40df84dafeb97e98bf1c, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:29.627510 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:29.637811 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1415/2821] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:29 2022] PUT /identity/v3/projects/bebf021bb9764059b6d1ef95ca0ca966/users/32320c57852b40df84dafeb97e98bf1c/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:30.271894 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfafb6a-28b0-4273-a4a0-082b2277688f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:30.272688 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfafb6a-28b0-4273-a4a0-082b2277688f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:30.273068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfafb6a-28b0-4273-a4a0-082b2277688f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:30.315645 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4cfafb6a-28b0-4273-a4a0-082b2277688f None None] MFA Rules not processed for user `32320c57852b40df84dafeb97e98bf1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:30.316924 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4cfafb6a-28b0-4273-a4a0-082b2277688f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:30.393923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1407/2822] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:30 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:30.833177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:30.835750 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:30.943061 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5446edab-dcb8-4cee-8928-53ce8e794787 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:30.949158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:30.949763 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:30.950044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:30.950317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:30.952098 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:30.965527 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:30.967228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:30.977371 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1416/2823] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:30 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:31.431985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfafb6a-28b0-4273-a4a0-082b2277688f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:31.494605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfafb6a-28b0-4273-a4a0-082b2277688f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:31.499088 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:31.499737 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:31.500021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:31.500295 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] PATH_INFO: `/v3/users/c50f032f8bc54f0d89cbd308a7bba511` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:31.501403 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c50f032f8bc54f0d89cbd308a7bba511)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:31.503317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:31.525219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:31.550122 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c50f032f8bc54f0d89cbd308a7bba511'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:31.557968 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c50f032f8bc54f0d89cbd308a7bba511'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:31.558605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c50f032f8bc54f0d89cbd308a7bba511'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:31.559237 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c50f032f8bc54f0d89cbd308a7bba511'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:31.566160 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c50f032f8bc54f0d89cbd308a7bba511'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:31.584388 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1408/2824] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:31 2022] DELETE /identity/v3/users/c50f032f8bc54f0d89cbd308a7bba511 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:31.750912 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:31.761837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb45082-262d-4301-a1a5-bfc9378cb52b service placement] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:31.919080 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:31.944488 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:31.945021 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:31.945300 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:31.945571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] PATH_INFO: `/v3/projects/8ffaa93476c94ff9beabdacf24864a4d` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:31.946930 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ffaa93476c94ff9beabdacf24864a4d)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:31.948858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:31.993458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ffaa93476c94ff9beabdacf24864a4d', 'request_id': 'req-52696750-050f-4b4e-825b-09d008d4f10f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.000046 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ffaa93476c94ff9beabdacf24864a4d', 'request_id': 'req-52696750-050f-4b4e-825b-09d008d4f10f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.005435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ffaa93476c94ff9beabdacf24864a4d', 'request_id': 'req-52696750-050f-4b4e-825b-09d008d4f10f'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.016879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] The token cache is being invalidate because project 8ffaa93476c94ff9beabdacf24864a4d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:32.017165 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.017775 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.019237 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1417/2825] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:31 2022] DELETE /identity/v3/projects/8ffaa93476c94ff9beabdacf24864a4d => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:32.024074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:32.025650 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:32.039823 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2a981-6fa5-4a6c-bb76-63a004ebb343 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:32.097335 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:32.097970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.098291 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.098630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] PATH_INFO: `/v3/users/32320c57852b40df84dafeb97e98bf1c` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.099900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] RBAC: Authorizing `identity:delete_user(user_id=32320c57852b40df84dafeb97e98bf1c)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:32.101914 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:32.147565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32320c57852b40df84dafeb97e98bf1c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.154848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32320c57852b40df84dafeb97e98bf1c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.159495 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32320c57852b40df84dafeb97e98bf1c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.160154 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32320c57852b40df84dafeb97e98bf1c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.167213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-f4380372-e540-4bc6-9cd0-6bcaecbf9f83 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32320c57852b40df84dafeb97e98bf1c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.184705 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1409/2826] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:32 2022] DELETE /identity/v3/users/32320c57852b40df84dafeb97e98bf1c => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:32.356356 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:32.358135 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:32.372133 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52696750-050f-4b4e-825b-09d008d4f10f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:32.430927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:32.431556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.431866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.432140 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] PATH_INFO: `/v3/projects/bebf021bb9764059b6d1ef95ca0ca966` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.433245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] RBAC: Authorizing `identity:delete_project(project_id=bebf021bb9764059b6d1ef95ca0ca966)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:32.435149 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:32.484579 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bebf021bb9764059b6d1ef95ca0ca966', 'request_id': 'req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.491915 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bebf021bb9764059b6d1ef95ca0ca966', 'request_id': 'req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.497293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bebf021bb9764059b6d1ef95ca0ca966', 'request_id': 'req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.499087 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:32.511649 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] The token cache is being invalidate because project bebf021bb9764059b6d1ef95ca0ca966 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:32.512001 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.512662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7cb4ae5b-79a8-4f54-9074-cd6b7431056d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:32.514132 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1418/2827] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:32 2022] DELETE /identity/v3/projects/bebf021bb9764059b6d1ef95ca0ca966 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:32.519020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab8e24e-a5e8-4575-8f8a-983f03043e8f None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.519498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab8e24e-a5e8-4575-8f8a-983f03043e8f None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.519602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab8e24e-a5e8-4575-8f8a-983f03043e8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.541271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4ab8e24e-a5e8-4575-8f8a-983f03043e8f None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:32.542450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4ab8e24e-a5e8-4575-8f8a-983f03043e8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:32.551955 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:DELETE=18,keystone:SELECT=80,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:32.604365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1410/2828] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:32.613494 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105be19-8e8e-4f5f-bb5e-5e1292905fdd None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.613780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105be19-8e8e-4f5f-bb5e-5e1292905fdd None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.614053 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105be19-8e8e-4f5f-bb5e-5e1292905fdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.633450 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-a105be19-8e8e-4f5f-bb5e-5e1292905fdd None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:32.634658 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-a105be19-8e8e-4f5f-bb5e-5e1292905fdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:32.642625 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1419/2829] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:28:32.647389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab8e24e-a5e8-4575-8f8a-983f03043e8f None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:32.657028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab8e24e-a5e8-4575-8f8a-983f03043e8f None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:32.661542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:32.662761 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.663039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.663306 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.663974 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:28:32.664271 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:32.665837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:32.677322 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1411/2830] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:28:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:28:32.681584 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a105be19-8e8e-4f5f-bb5e-5e1292905fdd None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:32.690796 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a105be19-8e8e-4f5f-bb5e-5e1292905fdd None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:32.696624 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18257980-ffd7-4640-9fe2-f82a5b4e6421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:32.697219 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18257980-ffd7-4640-9fe2-f82a5b4e6421 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.697498 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18257980-ffd7-4640-9fe2-f82a5b4e6421 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.697767 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18257980-ffd7-4640-9fe2-f82a5b4e6421 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.698394 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18257980-ffd7-4640-9fe2-f82a5b4e6421 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:32.700351 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18257980-ffd7-4640-9fe2-f82a5b4e6421 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:32.728646 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1420/2831] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:32 2022] POST /identity/v3/projects => generated 363 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:32.732669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:32.742689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7e5392-da73-4d2a-acc7-5b93fd8a3828 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:32.797772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044ea541-44ee-4a54-bf32-56199e7a8825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:32.798375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044ea541-44ee-4a54-bf32-56199e7a8825 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.798660 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044ea541-44ee-4a54-bf32-56199e7a8825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.798953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044ea541-44ee-4a54-bf32-56199e7a8825 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.799593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044ea541-44ee-4a54-bf32-56199e7a8825 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:32.801603 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044ea541-44ee-4a54-bf32-56199e7a8825 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:32.825090 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1412/2832] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:32 2022] POST /identity/v3/users => generated 282 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:32.829266 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18257980-ffd7-4640-9fe2-f82a5b4e6421 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:32.839656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18257980-ffd7-4640-9fe2-f82a5b4e6421 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:32.844302 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29913ac-fb09-4a87-8b35-2ed88142871a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:32.844899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29913ac-fb09-4a87-8b35-2ed88142871a admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.845177 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29913ac-fb09-4a87-8b35-2ed88142871a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.845451 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29913ac-fb09-4a87-8b35-2ed88142871a admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.846094 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29913ac-fb09-4a87-8b35-2ed88142871a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:32.847707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29913ac-fb09-4a87-8b35-2ed88142871a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:32.858002 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1421/2833] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:32.861996 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044ea541-44ee-4a54-bf32-56199e7a8825 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:32.872521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044ea541-44ee-4a54-bf32-56199e7a8825 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:32.877066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd711a69-d8a1-4fe2-916e-0b63bb2d7805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:32.877642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd711a69-d8a1-4fe2-916e-0b63bb2d7805 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.877918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd711a69-d8a1-4fe2-916e-0b63bb2d7805 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.878189 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd711a69-d8a1-4fe2-916e-0b63bb2d7805 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.878822 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd711a69-d8a1-4fe2-916e-0b63bb2d7805 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:32.880439 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd711a69-d8a1-4fe2-916e-0b63bb2d7805 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:32.891066 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1413/2834] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:32.895228 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29913ac-fb09-4a87-8b35-2ed88142871a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:32.905157 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29913ac-fb09-4a87-8b35-2ed88142871a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:32.909797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38228307-17bf-43fb-985e-213c55a7a38f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:32.910436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38228307-17bf-43fb-985e-213c55a7a38f admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.910720 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38228307-17bf-43fb-985e-213c55a7a38f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.910998 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38228307-17bf-43fb-985e-213c55a7a38f admin admin] PATH_INFO: `/v3/projects/df05ac923c704da5b5e8a4cb63077567/users/26f412f09df843afbdcda8edaed84b9b/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.925997 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38228307-17bf-43fb-985e-213c55a7a38f admin admin] RBAC: Authorizing `identity:create_grant(project_id=df05ac923c704da5b5e8a4cb63077567, user_id=26f412f09df843afbdcda8edaed84b9b, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:32.929475 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38228307-17bf-43fb-985e-213c55a7a38f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:32.942355 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1422/2835] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:32 2022] PUT /identity/v3/projects/df05ac923c704da5b5e8a4cb63077567/users/26f412f09df843afbdcda8edaed84b9b/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:28:32.952953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a6d41-ad54-449f-9e37-1cd94256555b None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:32.953236 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a6d41-ad54-449f-9e37-1cd94256555b None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:32.953510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a6d41-ad54-449f-9e37-1cd94256555b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:32.981451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-755a6d41-ad54-449f-9e37-1cd94256555b None None] MFA Rules not processed for user `26f412f09df843afbdcda8edaed84b9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:32.982638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-755a6d41-ad54-449f-9e37-1cd94256555b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:33.044572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1414/2836] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:32 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:28:33.048995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38228307-17bf-43fb-985e-213c55a7a38f admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:33.111280 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38228307-17bf-43fb-985e-213c55a7a38f admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:33.116005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac232a4-15da-44b7-9124-a4fc4eb43dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:33.116598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac232a4-15da-44b7-9124-a4fc4eb43dca admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:33.116879 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac232a4-15da-44b7-9124-a4fc4eb43dca admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:33.117146 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac232a4-15da-44b7-9124-a4fc4eb43dca admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:33.117764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac232a4-15da-44b7-9124-a4fc4eb43dca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:33.119651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac232a4-15da-44b7-9124-a4fc4eb43dca admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:33.145852 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1423/2837] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:33 2022] POST /identity/v3/projects => generated 363 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:33.150455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a6d41-ad54-449f-9e37-1cd94256555b None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:33.160852 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a6d41-ad54-449f-9e37-1cd94256555b None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:33.169571 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:33.220170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e23aa5e-9367-4e79-a392-6b4191c7213e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:33.220786 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e23aa5e-9367-4e79-a392-6b4191c7213e admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:33.221063 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e23aa5e-9367-4e79-a392-6b4191c7213e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:33.221334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e23aa5e-9367-4e79-a392-6b4191c7213e admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:33.221988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e23aa5e-9367-4e79-a392-6b4191c7213e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:33.224003 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e23aa5e-9367-4e79-a392-6b4191c7213e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:33.245405 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1415/2838] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:33 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:33.249556 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac232a4-15da-44b7-9124-a4fc4eb43dca admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:33.259916 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac232a4-15da-44b7-9124-a4fc4eb43dca admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:33.264571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5979a5b7-1822-4f08-8835-f5e5516c4de4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:33.265158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979a5b7-1822-4f08-8835-f5e5516c4de4 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:33.265435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979a5b7-1822-4f08-8835-f5e5516c4de4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:33.265707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979a5b7-1822-4f08-8835-f5e5516c4de4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:33.266339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5979a5b7-1822-4f08-8835-f5e5516c4de4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:33.267949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5979a5b7-1822-4f08-8835-f5e5516c4de4 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:33.278566 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1424/2839] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:33.283021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e23aa5e-9367-4e79-a392-6b4191c7213e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:33.293985 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e23aa5e-9367-4e79-a392-6b4191c7213e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:33.298613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760149d2-ff7b-4d04-beeb-b37dafe0aa4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:33.299205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760149d2-ff7b-4d04-beeb-b37dafe0aa4c admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:33.299484 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760149d2-ff7b-4d04-beeb-b37dafe0aa4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:33.299788 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760149d2-ff7b-4d04-beeb-b37dafe0aa4c admin admin] PATH_INFO: `/v3/projects/6f61557d64b64ef6b68a4940e565b67f/users/e9812b19d1524c2f8c1d10c046ca437a/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:33.314641 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760149d2-ff7b-4d04-beeb-b37dafe0aa4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f61557d64b64ef6b68a4940e565b67f, user_id=e9812b19d1524c2f8c1d10c046ca437a, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:33.317325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760149d2-ff7b-4d04-beeb-b37dafe0aa4c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:33.330619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1416/2840] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:33 2022] PUT /identity/v3/projects/6f61557d64b64ef6b68a4940e565b67f/users/e9812b19d1524c2f8c1d10c046ca437a/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:28:33.335282 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5979a5b7-1822-4f08-8835-f5e5516c4de4 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:33.345112 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5979a5b7-1822-4f08-8835-f5e5516c4de4 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:33.401417 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8e4c6-ffec-464a-a00d-a25716594cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:33.402017 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8e4c6-ffec-464a-a00d-a25716594cc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:33.402292 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8e4c6-ffec-464a-a00d-a25716594cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:33.402561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8e4c6-ffec-464a-a00d-a25716594cc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:33.403191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd8e4c6-ffec-464a-a00d-a25716594cc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:33.404821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd8e4c6-ffec-464a-a00d-a25716594cc3 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:33.414771 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1425/2841] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:33 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:33.418892 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760149d2-ff7b-4d04-beeb-b37dafe0aa4c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:33.428956 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760149d2-ff7b-4d04-beeb-b37dafe0aa4c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:33.433642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:33.434282 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:33.434556 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:33.434826 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] PATH_INFO: `/v3/projects/6f61557d64b64ef6b68a4940e565b67f/users/e9812b19d1524c2f8c1d10c046ca437a/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:33.436748 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f61557d64b64ef6b68a4940e565b67f, user_id=e9812b19d1524c2f8c1d10c046ca437a, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:33.439360 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:33.450158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1417/2842] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:33 2022] PUT /identity/v3/projects/6f61557d64b64ef6b68a4940e565b67f/users/e9812b19d1524c2f8c1d10c046ca437a/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:33.461253 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cac5a-bc03-4930-b52c-212b702fffb3 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:33.461533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cac5a-bc03-4930-b52c-212b702fffb3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:33.461802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cac5a-bc03-4930-b52c-212b702fffb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:33.468831 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:33.491922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-cc4cac5a-bc03-4930-b52c-212b702fffb3 None None] MFA Rules not processed for user `e9812b19d1524c2f8c1d10c046ca437a`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:33.493319 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-cc4cac5a-bc03-4930-b52c-212b702fffb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:33.565024 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1426/2843] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:33 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:33.577757 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:33.579637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:33.659399 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cfb4b6-c2ad-4822-ad55-a5d31ad97491 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:33.663272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a101d410-ab93-4ac8-96f0-5facf2111b3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:33.663899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a101d410-ab93-4ac8-96f0-5facf2111b3f service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:33.664179 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a101d410-ab93-4ac8-96f0-5facf2111b3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:33.664449 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a101d410-ab93-4ac8-96f0-5facf2111b3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:33.673125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a101d410-ab93-4ac8-96f0-5facf2111b3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:33.674534 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a101d410-ab93-4ac8-96f0-5facf2111b3f service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:33.684171 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1418/2844] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:33 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:34.091043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4cac5a-bc03-4930-b52c-212b702fffb3 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:34.152530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4cac5a-bc03-4930-b52c-212b702fffb3 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:34.156934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:34.157548 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:34.157824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:34.158097 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] PATH_INFO: `/v3/users/26f412f09df843afbdcda8edaed84b9b` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:34.159188 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] RBAC: Authorizing `identity:delete_user(user_id=26f412f09df843afbdcda8edaed84b9b)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:34.161122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:34.207056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26f412f09df843afbdcda8edaed84b9b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.215942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26f412f09df843afbdcda8edaed84b9b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.216591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26f412f09df843afbdcda8edaed84b9b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.217229 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26f412f09df843afbdcda8edaed84b9b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.224105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26f412f09df843afbdcda8edaed84b9b'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.241467 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1427/2845] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:34 2022] DELETE /identity/v3/users/26f412f09df843afbdcda8edaed84b9b => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:34.245858 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f09a04d-9ce9-4254-83ae-316b32e0fe3c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:34.246133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f09a04d-9ce9-4254-83ae-316b32e0fe3c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:34.246398 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f09a04d-9ce9-4254-83ae-316b32e0fe3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:34.265635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-7f09a04d-9ce9-4254-83ae-316b32e0fe3c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:34.266811 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7f09a04d-9ce9-4254-83ae-316b32e0fe3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:34.307869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:34.326838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1419/2846] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:34.337744 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:34.339116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:34.349051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:34.399678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4332dc19-77ca-4a7f-86d8-aacf368ff13e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:34.403400 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42e5524-c636-43f8-95ea-5f3281f7486a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:34.404044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e5524-c636-43f8-95ea-5f3281f7486a service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:34.404323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e5524-c636-43f8-95ea-5f3281f7486a service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:34.404596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e5524-c636-43f8-95ea-5f3281f7486a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:34.413923 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42e5524-c636-43f8-95ea-5f3281f7486a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:34.415350 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42e5524-c636-43f8-95ea-5f3281f7486a service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:34.424656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1428/2847] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:34.687224 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f09a04d-9ce9-4254-83ae-316b32e0fe3c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:34.688766 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7f09a04d-9ce9-4254-83ae-316b32e0fe3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:34.702683 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f09a04d-9ce9-4254-83ae-316b32e0fe3c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:34.707590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:34.708472 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:34.708752 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:34.709631 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] PATH_INFO: `/v3/projects/df05ac923c704da5b5e8a4cb63077567` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:34.710803 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] RBAC: Authorizing `identity:delete_project(project_id=df05ac923c704da5b5e8a4cb63077567)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:34.712847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:34.760426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df05ac923c704da5b5e8a4cb63077567', 'request_id': 'req-d02dd735-12f8-4434-9aed-d00275606fce'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.767395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df05ac923c704da5b5e8a4cb63077567', 'request_id': 'req-d02dd735-12f8-4434-9aed-d00275606fce'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.773939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df05ac923c704da5b5e8a4cb63077567', 'request_id': 'req-d02dd735-12f8-4434-9aed-d00275606fce'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.785064 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] The token cache is being invalidate because project df05ac923c704da5b5e8a4cb63077567 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:34.785350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.785981 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.788329 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1420/2848] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:34 2022] DELETE /identity/v3/projects/df05ac923c704da5b5e8a4cb63077567 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:34.792345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42e5524-c636-43f8-95ea-5f3281f7486a service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:34.801884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42e5524-c636-43f8-95ea-5f3281f7486a service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:34.859414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:34.860073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:34.860353 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:34.860626 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] PATH_INFO: `/v3/users/e9812b19d1524c2f8c1d10c046ca437a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:34.861736 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9812b19d1524c2f8c1d10c046ca437a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:34.863731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:34.907596 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9812b19d1524c2f8c1d10c046ca437a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.915777 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9812b19d1524c2f8c1d10c046ca437a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.916462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9812b19d1524c2f8c1d10c046ca437a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.917127 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9812b19d1524c2f8c1d10c046ca437a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.923980 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-99313f94-1dce-4b8a-b425-a4685be3ec0f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9812b19d1524c2f8c1d10c046ca437a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:34.940718 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1429/2849] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:34 2022] DELETE /identity/v3/users/e9812b19d1524c2f8c1d10c046ca437a => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:35.203961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:35.205478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:35.218840 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02dd735-12f8-4434-9aed-d00275606fce admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:35.275746 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:35.276375 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.276656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.276927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] PATH_INFO: `/v3/projects/6f61557d64b64ef6b68a4940e565b67f` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.278021 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f61557d64b64ef6b68a4940e565b67f)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:35.279944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:35.326652 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f61557d64b64ef6b68a4940e565b67f', 'request_id': 'req-107f6832-e926-4256-9d53-736e3851d83c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:35.333885 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f61557d64b64ef6b68a4940e565b67f', 'request_id': 'req-107f6832-e926-4256-9d53-736e3851d83c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:35.339658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f61557d64b64ef6b68a4940e565b67f', 'request_id': 'req-107f6832-e926-4256-9d53-736e3851d83c'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:35.347537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:35.352253 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] The token cache is being invalidate because project 6f61557d64b64ef6b68a4940e565b67f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:35.352614 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:35.353497 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-107f6832-e926-4256-9d53-736e3851d83c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:35.355605 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1421/2850] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:35 2022] DELETE /identity/v3/projects/6f61557d64b64ef6b68a4940e565b67f => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:35.360506 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f6d124-ed64-42a6-9e53-c58c096085e8 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.360994 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f6d124-ed64-42a6-9e53-c58c096085e8 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.361100 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f6d124-ed64-42a6-9e53-c58c096085e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.384044 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e2f6d124-ed64-42a6-9e53-c58c096085e8 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:35.385171 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e2f6d124-ed64-42a6-9e53-c58c096085e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:35.409235 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:35.444901 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1430/2851] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:35.506074 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7fd25-94ef-4ca1-bf10-c7f2576641cc None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.506454 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7fd25-94ef-4ca1-bf10-c7f2576641cc None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.506738 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c7fd25-94ef-4ca1-bf10-c7f2576641cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.526642 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-39c7fd25-94ef-4ca1-bf10-c7f2576641cc None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:35.527870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-39c7fd25-94ef-4ca1-bf10-c7f2576641cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:35.535888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1422/2852] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:35.541118 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f6d124-ed64-42a6-9e53-c58c096085e8 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:35.550670 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f6d124-ed64-42a6-9e53-c58c096085e8 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:35.555563 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:35.556183 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.556458 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.556726 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.557367 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:28:35.557666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:35.559240 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:35.570773 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1431/2853] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:28:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:28:35.575231 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c7fd25-94ef-4ca1-bf10-c7f2576641cc None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:35.585123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c7fd25-94ef-4ca1-bf10-c7f2576641cc None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:35.589864 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee54d1-3851-457f-b02d-bb9e5e9536f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:35.590457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee54d1-3851-457f-b02d-bb9e5e9536f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.590736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee54d1-3851-457f-b02d-bb9e5e9536f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.591009 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee54d1-3851-457f-b02d-bb9e5e9536f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.591653 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee54d1-3851-457f-b02d-bb9e5e9536f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:35.593573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee54d1-3851-457f-b02d-bb9e5e9536f5 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:35.620855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1423/2854] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:35 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:28:35.625082 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:35.634969 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219735ae-a563-4918-bf9b-19f0dcf6dd2b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:35.689438 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d0ee4-5ead-4bd8-b9c5-268a453bc62a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:35.690032 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d0ee4-5ead-4bd8-b9c5-268a453bc62a admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.690308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d0ee4-5ead-4bd8-b9c5-268a453bc62a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.690595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d0ee4-5ead-4bd8-b9c5-268a453bc62a admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.691198 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d0ee4-5ead-4bd8-b9c5-268a453bc62a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:35.693134 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d0ee4-5ead-4bd8-b9c5-268a453bc62a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:35.714630 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1432/2855] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:35 2022] POST /identity/v3/users => generated 294 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:35.718912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee54d1-3851-457f-b02d-bb9e5e9536f5 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:35.729365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee54d1-3851-457f-b02d-bb9e5e9536f5 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:35.734005 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7bd2ab-7fc2-4029-863b-b74f267db880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:35.734602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7bd2ab-7fc2-4029-863b-b74f267db880 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.734883 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7bd2ab-7fc2-4029-863b-b74f267db880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.735157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7bd2ab-7fc2-4029-863b-b74f267db880 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.735838 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7bd2ab-7fc2-4029-863b-b74f267db880 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:35.737434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7bd2ab-7fc2-4029-863b-b74f267db880 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:35.747228 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1424/2856] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:35.751126 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d0ee4-5ead-4bd8-b9c5-268a453bc62a admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:35.761639 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d0ee4-5ead-4bd8-b9c5-268a453bc62a admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:35.766236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35b6573-9eba-4fdf-823f-1f3179570799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:35.766827 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b6573-9eba-4fdf-823f-1f3179570799 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.767109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b6573-9eba-4fdf-823f-1f3179570799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.767383 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b6573-9eba-4fdf-823f-1f3179570799 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.768085 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35b6573-9eba-4fdf-823f-1f3179570799 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:35.769668 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35b6573-9eba-4fdf-823f-1f3179570799 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:35.780532 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1433/2857] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:35.784590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7bd2ab-7fc2-4029-863b-b74f267db880 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:35.794731 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7bd2ab-7fc2-4029-863b-b74f267db880 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:35.799313 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25245890-db70-40c6-b1d2-509b24cb171c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:35.799970 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25245890-db70-40c6-b1d2-509b24cb171c admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.800254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25245890-db70-40c6-b1d2-509b24cb171c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.800530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25245890-db70-40c6-b1d2-509b24cb171c admin admin] PATH_INFO: `/v3/projects/8d50c5fc95f741febe0d9eabb3008a91/users/77138d31999649c4aa6bb8c77e26e680/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.814894 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25245890-db70-40c6-b1d2-509b24cb171c admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d50c5fc95f741febe0d9eabb3008a91, user_id=77138d31999649c4aa6bb8c77e26e680, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:35.817590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25245890-db70-40c6-b1d2-509b24cb171c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:35.831086 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1425/2858] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:35 2022] PUT /identity/v3/projects/8d50c5fc95f741febe0d9eabb3008a91/users/77138d31999649c4aa6bb8c77e26e680/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:35.841662 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c207c1-f83c-4b55-8328-d07f9442bcad None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:35.841954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c207c1-f83c-4b55-8328-d07f9442bcad None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:35.842230 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c207c1-f83c-4b55-8328-d07f9442bcad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:35.870549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-d3c207c1-f83c-4b55-8328-d07f9442bcad None None] MFA Rules not processed for user `77138d31999649c4aa6bb8c77e26e680`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:35.871800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d3c207c1-f83c-4b55-8328-d07f9442bcad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:35.934533 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1434/2859] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:35 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:35.938890 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25245890-db70-40c6-b1d2-509b24cb171c admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:36.001251 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25245890-db70-40c6-b1d2-509b24cb171c admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:36.005944 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91538b21-84bd-425c-b04d-b4857600a9f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:36.006717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91538b21-84bd-425c-b04d-b4857600a9f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:36.007001 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91538b21-84bd-425c-b04d-b4857600a9f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:36.007278 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91538b21-84bd-425c-b04d-b4857600a9f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:36.007933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91538b21-84bd-425c-b04d-b4857600a9f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:36.009833 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91538b21-84bd-425c-b04d-b4857600a9f8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:36.035747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1426/2860] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:35 2022] POST /identity/v3/projects => generated 387 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:36.040368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c207c1-f83c-4b55-8328-d07f9442bcad None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:36.050944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c207c1-f83c-4b55-8328-d07f9442bcad None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:36.070722 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:36.109250 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09735def-c865-4e4e-8b5c-3fb2671c186e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:36.109861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09735def-c865-4e4e-8b5c-3fb2671c186e admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:36.110142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09735def-c865-4e4e-8b5c-3fb2671c186e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:36.110416 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09735def-c865-4e4e-8b5c-3fb2671c186e admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:36.111055 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09735def-c865-4e4e-8b5c-3fb2671c186e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:36.113018 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09735def-c865-4e4e-8b5c-3fb2671c186e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:36.135309 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1435/2861] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:36 2022] POST /identity/v3/users => generated 300 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:36.139286 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91538b21-84bd-425c-b04d-b4857600a9f8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:36.150625 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91538b21-84bd-425c-b04d-b4857600a9f8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:36.155263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba343845-2432-493b-9391-c6df7172a7da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:36.155900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba343845-2432-493b-9391-c6df7172a7da admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:36.156185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba343845-2432-493b-9391-c6df7172a7da admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:36.156461 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba343845-2432-493b-9391-c6df7172a7da admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:36.157104 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba343845-2432-493b-9391-c6df7172a7da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:36.158734 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba343845-2432-493b-9391-c6df7172a7da admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:36.169215 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1427/2862] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:36.173577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09735def-c865-4e4e-8b5c-3fb2671c186e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:36.184175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09735def-c865-4e4e-8b5c-3fb2671c186e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:36.188761 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae610d7-e794-4119-b3f3-9e10f939c70d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:36.189339 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae610d7-e794-4119-b3f3-9e10f939c70d admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:36.189618 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae610d7-e794-4119-b3f3-9e10f939c70d admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:36.189893 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae610d7-e794-4119-b3f3-9e10f939c70d admin admin] PATH_INFO: `/v3/projects/53ca28271ce84c45a7a4e23a9ede1bbd/users/afc2106c5a3a418a9f5035bca292be00/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:36.205262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae610d7-e794-4119-b3f3-9e10f939c70d admin admin] RBAC: Authorizing `identity:create_grant(project_id=53ca28271ce84c45a7a4e23a9ede1bbd, user_id=afc2106c5a3a418a9f5035bca292be00, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:36.207941 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae610d7-e794-4119-b3f3-9e10f939c70d admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:36.220678 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1436/2863] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:36 2022] PUT /identity/v3/projects/53ca28271ce84c45a7a4e23a9ede1bbd/users/afc2106c5a3a418a9f5035bca292be00/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:36.224865 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba343845-2432-493b-9391-c6df7172a7da admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:36.234580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba343845-2432-493b-9391-c6df7172a7da admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:36.290542 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203b4db8-c1f6-4ac2-81a3-35b2b7bd2996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:36.291143 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203b4db8-c1f6-4ac2-81a3-35b2b7bd2996 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:36.291427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203b4db8-c1f6-4ac2-81a3-35b2b7bd2996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:36.291724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203b4db8-c1f6-4ac2-81a3-35b2b7bd2996 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:36.292377 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203b4db8-c1f6-4ac2-81a3-35b2b7bd2996 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:36.293961 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203b4db8-c1f6-4ac2-81a3-35b2b7bd2996 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:36.304717 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1428/2864] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:36 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:28:36.308872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae610d7-e794-4119-b3f3-9e10f939c70d admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:36.320643 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae610d7-e794-4119-b3f3-9e10f939c70d admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:36.325562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:36.326256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:36.326537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:36.326814 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] PATH_INFO: `/v3/projects/53ca28271ce84c45a7a4e23a9ede1bbd/users/afc2106c5a3a418a9f5035bca292be00/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:36.328807 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53ca28271ce84c45a7a4e23a9ede1bbd, user_id=afc2106c5a3a418a9f5035bca292be00, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:36.331422 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:36.341797 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1437/2865] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:36 2022] PUT /identity/v3/projects/53ca28271ce84c45a7a4e23a9ede1bbd/users/afc2106c5a3a418a9f5035bca292be00/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:36.352048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f21b988-2809-4cca-9b6e-385008c58278 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:36.352334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f21b988-2809-4cca-9b6e-385008c58278 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:36.352637 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f21b988-2809-4cca-9b6e-385008c58278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:36.366939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:36.382273 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-6f21b988-2809-4cca-9b6e-385008c58278 None None] MFA Rules not processed for user `afc2106c5a3a418a9f5035bca292be00`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:36.383648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6f21b988-2809-4cca-9b6e-385008c58278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:36.452530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1429/2866] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:36 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:36.463156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:36.464906 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:36.522156 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc2d599-1a8d-4aca-a7d4-431c2b860169 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:36.525849 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76c0a46-383e-4492-ab15-4c3812742a00 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:36.526439 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c0a46-383e-4492-ab15-4c3812742a00 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:36.526717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c0a46-383e-4492-ab15-4c3812742a00 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:36.526990 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c0a46-383e-4492-ab15-4c3812742a00 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:36.535853 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76c0a46-383e-4492-ab15-4c3812742a00 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:36.537779 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76c0a46-383e-4492-ab15-4c3812742a00 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:36.547172 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1438/2867] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:36 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:37.262960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f21b988-2809-4cca-9b6e-385008c58278 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:37.326028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f21b988-2809-4cca-9b6e-385008c58278 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:37.330471 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:37.331130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:37.331411 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:37.331780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] PATH_INFO: `/v3/users/77138d31999649c4aa6bb8c77e26e680` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:37.332922 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] RBAC: Authorizing `identity:delete_user(user_id=77138d31999649c4aa6bb8c77e26e680)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:37.335759 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:37.383594 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77138d31999649c4aa6bb8c77e26e680'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.390613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77138d31999649c4aa6bb8c77e26e680'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.391284 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77138d31999649c4aa6bb8c77e26e680'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.391939 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77138d31999649c4aa6bb8c77e26e680'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.399382 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77138d31999649c4aa6bb8c77e26e680'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.416267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1430/2868] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:37 2022] DELETE /identity/v3/users/77138d31999649c4aa6bb8c77e26e680 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:37.420804 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfebbd7a-63a3-4c4d-9a84-8e1dc9abef59 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:37.421086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfebbd7a-63a3-4c4d-9a84-8e1dc9abef59 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:37.421359 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfebbd7a-63a3-4c4d-9a84-8e1dc9abef59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:37.439883 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-bfebbd7a-63a3-4c4d-9a84-8e1dc9abef59 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:37.441073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bfebbd7a-63a3-4c4d-9a84-8e1dc9abef59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:37.499047 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1439/2869] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:37.509272 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:37.510689 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:37.524569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:37.570875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe6260d-4253-4c73-adc8-36a229b8c48e admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:37.574791 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39dc1c-c36c-4f31-8636-0913a3dc6718 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:37.575395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39dc1c-c36c-4f31-8636-0913a3dc6718 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:37.575701 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39dc1c-c36c-4f31-8636-0913a3dc6718 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:37.575979 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39dc1c-c36c-4f31-8636-0913a3dc6718 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:37.585444 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39dc1c-c36c-4f31-8636-0913a3dc6718 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:37.586848 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39dc1c-c36c-4f31-8636-0913a3dc6718 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:37.596359 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1431/2870] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:37.865030 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfebbd7a-63a3-4c4d-9a84-8e1dc9abef59 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:37.866539 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bfebbd7a-63a3-4c4d-9a84-8e1dc9abef59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:37.880204 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfebbd7a-63a3-4c4d-9a84-8e1dc9abef59 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:37.885435 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:37.886306 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:37.886590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:37.886866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] PATH_INFO: `/v3/projects/8d50c5fc95f741febe0d9eabb3008a91` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:37.888068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d50c5fc95f741febe0d9eabb3008a91)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:37.890051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:37.906592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:37.937308 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d50c5fc95f741febe0d9eabb3008a91', 'request_id': 'req-f54356da-d9d7-49e2-9051-263006c5b064'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.944719 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d50c5fc95f741febe0d9eabb3008a91', 'request_id': 'req-f54356da-d9d7-49e2-9051-263006c5b064'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.950536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d50c5fc95f741febe0d9eabb3008a91', 'request_id': 'req-f54356da-d9d7-49e2-9051-263006c5b064'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.962116 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] The token cache is being invalidate because project 8d50c5fc95f741febe0d9eabb3008a91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:37.962402 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.963025 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:37.964521 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1440/2871] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:37 2022] DELETE /identity/v3/projects/8d50c5fc95f741febe0d9eabb3008a91 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:37.968465 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39dc1c-c36c-4f31-8636-0913a3dc6718 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:37.979007 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39dc1c-c36c-4f31-8636-0913a3dc6718 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:38.034485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:38.035109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.035395 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.035669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] PATH_INFO: `/v3/users/afc2106c5a3a418a9f5035bca292be00` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.036821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=afc2106c5a3a418a9f5035bca292be00)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:38.039367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:38.084515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afc2106c5a3a418a9f5035bca292be00'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.091805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afc2106c5a3a418a9f5035bca292be00'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.092415 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afc2106c5a3a418a9f5035bca292be00'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.093017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afc2106c5a3a418a9f5035bca292be00'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.100343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-98fd9538-eb1f-4263-91db-e5083ae8db9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afc2106c5a3a418a9f5035bca292be00'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.118891 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1432/2872] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:37 2022] DELETE /identity/v3/users/afc2106c5a3a418a9f5035bca292be00 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:38.392725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:38.394283 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:38.409965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54356da-d9d7-49e2-9051-263006c5b064 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:38.470323 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:38.470946 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.471223 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.471493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] PATH_INFO: `/v3/projects/53ca28271ce84c45a7a4e23a9ede1bbd` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.472619 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] RBAC: Authorizing `identity:delete_project(project_id=53ca28271ce84c45a7a4e23a9ede1bbd)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:38.474508 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:38.521526 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53ca28271ce84c45a7a4e23a9ede1bbd', 'request_id': 'req-93231f00-8dfb-47f1-a81e-40f510c8658a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.528959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53ca28271ce84c45a7a4e23a9ede1bbd', 'request_id': 'req-93231f00-8dfb-47f1-a81e-40f510c8658a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.534694 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53ca28271ce84c45a7a4e23a9ede1bbd', 'request_id': 'req-93231f00-8dfb-47f1-a81e-40f510c8658a'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.545707 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] The token cache is being invalidate because project 53ca28271ce84c45a7a4e23a9ede1bbd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:38.545996 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.546621 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-93231f00-8dfb-47f1-a81e-40f510c8658a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:38.548099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1441/2873] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:38 2022] DELETE /identity/v3/projects/53ca28271ce84c45a7a4e23a9ede1bbd => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:38.552916 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0faba2-3e2e-49d8-82ad-5085b630d74e None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.553381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0faba2-3e2e-49d8-82ad-5085b630d74e None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.553485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0faba2-3e2e-49d8-82ad-5085b630d74e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.574530 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-4c0faba2-3e2e-49d8-82ad-5085b630d74e None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:38.575739 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-4c0faba2-3e2e-49d8-82ad-5085b630d74e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:38.606325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:38.638563 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1433/2874] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:38.646236 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32da929-5580-42e5-9c40-ba697f7fa8f6 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.646524 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32da929-5580-42e5-9c40-ba697f7fa8f6 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.646800 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32da929-5580-42e5-9c40-ba697f7fa8f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.666949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-e32da929-5580-42e5-9c40-ba697f7fa8f6 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:38.668195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-e32da929-5580-42e5-9c40-ba697f7fa8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:38.676175 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1442/2875] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:38.681006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0faba2-3e2e-49d8-82ad-5085b630d74e None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:38.690685 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0faba2-3e2e-49d8-82ad-5085b630d74e None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:38.695325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:38.695945 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.696490 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.696760 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.697426 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:28:38.697726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:38.699299 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:38.710628 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1434/2876] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:28:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:28:38.714866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32da929-5580-42e5-9c40-ba697f7fa8f6 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:38.723962 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32da929-5580-42e5-9c40-ba697f7fa8f6 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:38.729256 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7475b-71a1-4fac-a20b-6608a6fa51c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:38.729844 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7475b-71a1-4fac-a20b-6608a6fa51c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.730123 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7475b-71a1-4fac-a20b-6608a6fa51c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.730395 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7475b-71a1-4fac-a20b-6608a6fa51c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.731014 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc7475b-71a1-4fac-a20b-6608a6fa51c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:38.732944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc7475b-71a1-4fac-a20b-6608a6fa51c4 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:38.760612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1443/2877] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:38 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:38.765133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:38.775217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81faa09-54c8-4780-b1cd-dc9bbb2650a5 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:38.838849 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f648e9-1312-4d53-bab1-fee0e66af982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:38.839450 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f648e9-1312-4d53-bab1-fee0e66af982 admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.839756 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f648e9-1312-4d53-bab1-fee0e66af982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.840030 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f648e9-1312-4d53-bab1-fee0e66af982 admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.840654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f648e9-1312-4d53-bab1-fee0e66af982 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:38.842567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f648e9-1312-4d53-bab1-fee0e66af982 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:38.864075 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1435/2878] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:38 2022] POST /identity/v3/users => generated 288 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:28:38.868967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7475b-71a1-4fac-a20b-6608a6fa51c4 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:38.875598 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:38.879595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7475b-71a1-4fac-a20b-6608a6fa51c4 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:38.885872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd40342-f3b0-44c2-9a22-dc756d23d753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:38.886709 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd40342-f3b0-44c2-9a22-dc756d23d753 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.886999 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd40342-f3b0-44c2-9a22-dc756d23d753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.887428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd40342-f3b0-44c2-9a22-dc756d23d753 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.888258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd40342-f3b0-44c2-9a22-dc756d23d753 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:38.889922 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd40342-f3b0-44c2-9a22-dc756d23d753 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:38.899896 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1444/2879] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:38.903923 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f648e9-1312-4d53-bab1-fee0e66af982 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:38.914560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f648e9-1312-4d53-bab1-fee0e66af982 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:38.919339 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7493fa98-ed62-499a-8e30-605f09e38845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:38.919962 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7493fa98-ed62-499a-8e30-605f09e38845 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.920243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7493fa98-ed62-499a-8e30-605f09e38845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.920537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7493fa98-ed62-499a-8e30-605f09e38845 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.921172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7493fa98-ed62-499a-8e30-605f09e38845 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:38.922777 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7493fa98-ed62-499a-8e30-605f09e38845 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:38.933920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1436/2880] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:38.937742 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd40342-f3b0-44c2-9a22-dc756d23d753 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:38.949428 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd40342-f3b0-44c2-9a22-dc756d23d753 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:38.953950 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:38.954590 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.954863 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.955131 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] PATH_INFO: `/v3/projects/92b7a703ac864d19b86bff4831781ea9/users/6fff85bae5454c0cb8e45f19d4646bd1/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:38.969837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92b7a703ac864d19b86bff4831781ea9, user_id=6fff85bae5454c0cb8e45f19d4646bd1, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:38.972568 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:38.986058 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1445/2881] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:38 2022] PUT /identity/v3/projects/92b7a703ac864d19b86bff4831781ea9/users/6fff85bae5454c0cb8e45f19d4646bd1/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:38.996304 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc6789-9475-4f4a-a8fa-cdf145f4dde0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:38.996592 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc6789-9475-4f4a-a8fa-cdf145f4dde0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:38.996866 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc6789-9475-4f4a-a8fa-cdf145f4dde0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:39.024868 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-d8bc6789-9475-4f4a-a8fa-cdf145f4dde0 None None] MFA Rules not processed for user `6fff85bae5454c0cb8e45f19d4646bd1`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:39.026077 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-d8bc6789-9475-4f4a-a8fa-cdf145f4dde0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:39.089100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1437/2882] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:38 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:39.100397 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:39.102713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:39.157966 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab8a296-b33e-4f47-81b0-7b547e0a93f8 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:39.161729 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcec570-6350-49bb-bf3e-6ab99eb61d46 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:39.162335 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcec570-6350-49bb-bf3e-6ab99eb61d46 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:39.162614 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcec570-6350-49bb-bf3e-6ab99eb61d46 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:39.162885 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcec570-6350-49bb-bf3e-6ab99eb61d46 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:39.172185 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcec570-6350-49bb-bf3e-6ab99eb61d46 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:39.173603 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcec570-6350-49bb-bf3e-6ab99eb61d46 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:39.182794 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1446/2883] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:39 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:42.197330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bc6789-9475-4f4a-a8fa-cdf145f4dde0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:42.229028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:42.259907 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bc6789-9475-4f4a-a8fa-cdf145f4dde0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:42.264541 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:42.265261 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:42.265537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:42.265806 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] PATH_INFO: `/v3/users/6fff85bae5454c0cb8e45f19d4646bd1` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:42.266899 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6fff85bae5454c0cb8e45f19d4646bd1)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:42.268827 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:42.315084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6fff85bae5454c0cb8e45f19d4646bd1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.322547 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6fff85bae5454c0cb8e45f19d4646bd1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.323195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6fff85bae5454c0cb8e45f19d4646bd1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.323869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6fff85bae5454c0cb8e45f19d4646bd1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.331096 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6fff85bae5454c0cb8e45f19d4646bd1'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.348381 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1438/2884] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:42 2022] DELETE /identity/v3/users/6fff85bae5454c0cb8e45f19d4646bd1 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:42.353396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df4fcf7-68bf-4ac3-b5ae-cfc98a126af3 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:42.353900 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df4fcf7-68bf-4ac3-b5ae-cfc98a126af3 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:42.354028 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df4fcf7-68bf-4ac3-b5ae-cfc98a126af3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:42.373152 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-2df4fcf7-68bf-4ac3-b5ae-cfc98a126af3 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:42.374365 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2df4fcf7-68bf-4ac3-b5ae-cfc98a126af3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:42.432489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1447/2885] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:28:42.442402 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:42.443867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:42.507148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea1491-d5a5-4e4f-a613-ad732e793516 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:42.510779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97521981-cb37-4d64-995b-16b1aa8f4b68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:42.511379 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97521981-cb37-4d64-995b-16b1aa8f4b68 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:42.511654 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97521981-cb37-4d64-995b-16b1aa8f4b68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:42.511951 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97521981-cb37-4d64-995b-16b1aa8f4b68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:42.520622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97521981-cb37-4d64-995b-16b1aa8f4b68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:42.522020 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97521981-cb37-4d64-995b-16b1aa8f4b68 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:42.532047 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1439/2886] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:42.800860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df4fcf7-68bf-4ac3-b5ae-cfc98a126af3 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:42.802271 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2df4fcf7-68bf-4ac3-b5ae-cfc98a126af3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:42.814764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df4fcf7-68bf-4ac3-b5ae-cfc98a126af3 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:42.820184 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:42.820824 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:42.821107 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:42.821638 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] PATH_INFO: `/v3/projects/92b7a703ac864d19b86bff4831781ea9` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:42.822753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] RBAC: Authorizing `identity:delete_project(project_id=92b7a703ac864d19b86bff4831781ea9)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:42.824753 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:42.870793 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92b7a703ac864d19b86bff4831781ea9', 'request_id': 'req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.878073 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92b7a703ac864d19b86bff4831781ea9', 'request_id': 'req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.885208 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92b7a703ac864d19b86bff4831781ea9', 'request_id': 'req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.887160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:42.898392 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] The token cache is being invalidate because project 92b7a703ac864d19b86bff4831781ea9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:42.898680 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.899317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-0923f93d-0c75-4e60-8c0c-43cf1b4d063c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:42.900841 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1448/2887] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:42 2022] DELETE /identity/v3/projects/92b7a703ac864d19b86bff4831781ea9 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:42.905880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03f507f-aa2f-4b96-b0ac-c6e9c1bc3d30 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:42.906169 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03f507f-aa2f-4b96-b0ac-c6e9c1bc3d30 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:42.906442 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03f507f-aa2f-4b96-b0ac-c6e9c1bc3d30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:42.925741 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-c03f507f-aa2f-4b96-b0ac-c6e9c1bc3d30 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:42.926928 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-c03f507f-aa2f-4b96-b0ac-c6e9c1bc3d30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:42.972028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:42.987955 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1440/2888] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:42.996959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bf2d8d-a7d3-4d1d-9a70-6d2dbe8af5e1 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:42.997245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bf2d8d-a7d3-4d1d-9a70-6d2dbe8af5e1 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:42.997518 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bf2d8d-a7d3-4d1d-9a70-6d2dbe8af5e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.017252 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-c5bf2d8d-a7d3-4d1d-9a70-6d2dbe8af5e1 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:43.018431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-c5bf2d8d-a7d3-4d1d-9a70-6d2dbe8af5e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:43.027079 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1449/2889] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:43.031208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03f507f-aa2f-4b96-b0ac-c6e9c1bc3d30 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.041498 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03f507f-aa2f-4b96-b0ac-c6e9c1bc3d30 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.046044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.046635 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.046915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.047186 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] PATH_INFO: `/v3/domains` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.047867 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81654) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:28:43.048170 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.049749 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.061099 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1441/2890] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:28:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:28:43.065406 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bf2d8d-a7d3-4d1d-9a70-6d2dbe8af5e1 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.075500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bf2d8d-a7d3-4d1d-9a70-6d2dbe8af5e1 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.080816 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f02b0f-b53d-4eb8-9ca9-f406260396c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.081457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f02b0f-b53d-4eb8-9ca9-f406260396c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.081914 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f02b0f-b53d-4eb8-9ca9-f406260396c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.082246 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f02b0f-b53d-4eb8-9ca9-f406260396c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.082872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f02b0f-b53d-4eb8-9ca9-f406260396c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.085633 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f02b0f-b53d-4eb8-9ca9-f406260396c2 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.113372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1450/2891] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:43 2022] POST /identity/v3/projects => generated 375 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:43.117706 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.127577 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df3ee45-d329-4d74-8805-a91b9622c974 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.182583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709af581-a556-46a6-9110-ca0964af59bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.183276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709af581-a556-46a6-9110-ca0964af59bc admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.183560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709af581-a556-46a6-9110-ca0964af59bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.183869 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709af581-a556-46a6-9110-ca0964af59bc admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.184494 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709af581-a556-46a6-9110-ca0964af59bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.186400 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709af581-a556-46a6-9110-ca0964af59bc admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.207918 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1442/2892] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:43 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 15 14:28:43.212666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f02b0f-b53d-4eb8-9ca9-f406260396c2 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.222995 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f02b0f-b53d-4eb8-9ca9-f406260396c2 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.227679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb857b34-5aae-40ae-8e16-db582e82adcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.228293 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb857b34-5aae-40ae-8e16-db582e82adcd admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.228576 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb857b34-5aae-40ae-8e16-db582e82adcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.228852 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb857b34-5aae-40ae-8e16-db582e82adcd admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.229493 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb857b34-5aae-40ae-8e16-db582e82adcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.231078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb857b34-5aae-40ae-8e16-db582e82adcd admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.240837 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1451/2893] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:43.245098 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709af581-a556-46a6-9110-ca0964af59bc admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.254893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709af581-a556-46a6-9110-ca0964af59bc admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.260344 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae07a843-e253-40ba-bd9b-001d3ac961c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.260937 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae07a843-e253-40ba-bd9b-001d3ac961c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.261218 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae07a843-e253-40ba-bd9b-001d3ac961c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.261491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae07a843-e253-40ba-bd9b-001d3ac961c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.262132 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae07a843-e253-40ba-bd9b-001d3ac961c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.263723 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae07a843-e253-40ba-bd9b-001d3ac961c8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.274821 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1443/2894] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:43.278859 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb857b34-5aae-40ae-8e16-db582e82adcd admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.289430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb857b34-5aae-40ae-8e16-db582e82adcd admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.294051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfeb132-8d4e-4257-8850-2fd013b94ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.294692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfeb132-8d4e-4257-8850-2fd013b94ef9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.294971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfeb132-8d4e-4257-8850-2fd013b94ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.295243 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfeb132-8d4e-4257-8850-2fd013b94ef9 admin admin] PATH_INFO: `/v3/projects/665f2b8696ce494188535f65b8900b1a/users/fff060c5b46a4d479cede3f35cc9a709/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.309748 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfeb132-8d4e-4257-8850-2fd013b94ef9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=665f2b8696ce494188535f65b8900b1a, user_id=fff060c5b46a4d479cede3f35cc9a709, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.312434 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfeb132-8d4e-4257-8850-2fd013b94ef9 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.326060 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1452/2895] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:43 2022] PUT /identity/v3/projects/665f2b8696ce494188535f65b8900b1a/users/fff060c5b46a4d479cede3f35cc9a709/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:43.337151 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16c79f0-5a71-419f-84be-e651976c9140 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.337434 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16c79f0-5a71-419f-84be-e651976c9140 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.337710 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16c79f0-5a71-419f-84be-e651976c9140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.364747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e16c79f0-5a71-419f-84be-e651976c9140 None None] MFA Rules not processed for user `fff060c5b46a4d479cede3f35cc9a709`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:43.365949 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e16c79f0-5a71-419f-84be-e651976c9140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:43.429455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1444/2896] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:28:43.434270 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfeb132-8d4e-4257-8850-2fd013b94ef9 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.495974 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfeb132-8d4e-4257-8850-2fd013b94ef9 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.500644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a22a2-104f-410f-aba8-afb5328e030e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.501239 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a22a2-104f-410f-aba8-afb5328e030e admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.501516 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a22a2-104f-410f-aba8-afb5328e030e admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.501802 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303a22a2-104f-410f-aba8-afb5328e030e admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.502443 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303a22a2-104f-410f-aba8-afb5328e030e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.504407 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303a22a2-104f-410f-aba8-afb5328e030e admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.530347 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1453/2897] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:43 2022] POST /identity/v3/projects => generated 377 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:43.534491 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16c79f0-5a71-419f-84be-e651976c9140 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.545202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16c79f0-5a71-419f-84be-e651976c9140 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.590324 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:43.606669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b410053-f56c-4327-a19e-f8015c7d049f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.607676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b410053-f56c-4327-a19e-f8015c7d049f admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.608219 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b410053-f56c-4327-a19e-f8015c7d049f admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.608682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b410053-f56c-4327-a19e-f8015c7d049f admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.609809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b410053-f56c-4327-a19e-f8015c7d049f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.612796 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b410053-f56c-4327-a19e-f8015c7d049f admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.635330 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1445/2898] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:43 2022] POST /identity/v3/users => generated 289 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:43.639378 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a22a2-104f-410f-aba8-afb5328e030e admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.649860 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303a22a2-104f-410f-aba8-afb5328e030e admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.654384 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1670732f-2ae9-4cb9-addb-aca17646f0ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.654973 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1670732f-2ae9-4cb9-addb-aca17646f0ba admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.655255 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1670732f-2ae9-4cb9-addb-aca17646f0ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.655525 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1670732f-2ae9-4cb9-addb-aca17646f0ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.656191 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1670732f-2ae9-4cb9-addb-aca17646f0ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.657772 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1670732f-2ae9-4cb9-addb-aca17646f0ba admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.668457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1454/2899] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:43.672539 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b410053-f56c-4327-a19e-f8015c7d049f admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.682820 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b410053-f56c-4327-a19e-f8015c7d049f admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.687319 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283d8fee-091d-456b-be09-228dfcea011c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.687938 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d8fee-091d-456b-be09-228dfcea011c admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.688216 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d8fee-091d-456b-be09-228dfcea011c admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.688486 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d8fee-091d-456b-be09-228dfcea011c admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.689217 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283d8fee-091d-456b-be09-228dfcea011c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.690893 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283d8fee-091d-456b-be09-228dfcea011c admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.701856 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1446/2900] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 15 14:28:43.705909 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1670732f-2ae9-4cb9-addb-aca17646f0ba admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.716128 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1670732f-2ae9-4cb9-addb-aca17646f0ba admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.720724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02907aa9-3a38-4dbe-bce4-8acb64b8d421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.721372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02907aa9-3a38-4dbe-bce4-8acb64b8d421 admin admin] REQUEST_METHOD: `PUT` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.721656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02907aa9-3a38-4dbe-bce4-8acb64b8d421 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.721928 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02907aa9-3a38-4dbe-bce4-8acb64b8d421 admin admin] PATH_INFO: `/v3/projects/8474325d898447e2aa45b1c8400ab6f7/users/733f529b645a40ff99245aed2991b626/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.736803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02907aa9-3a38-4dbe-bce4-8acb64b8d421 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8474325d898447e2aa45b1c8400ab6f7, user_id=733f529b645a40ff99245aed2991b626, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.739446 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02907aa9-3a38-4dbe-bce4-8acb64b8d421 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.753195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1455/2901] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:43 2022] PUT /identity/v3/projects/8474325d898447e2aa45b1c8400ab6f7/users/733f529b645a40ff99245aed2991b626/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:28:43.763953 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62e4d41-e632-4fc1-bd2a-11d3951c4b5a None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.764241 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62e4d41-e632-4fc1-bd2a-11d3951c4b5a None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.764521 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62e4d41-e632-4fc1-bd2a-11d3951c4b5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.792002 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-b62e4d41-e632-4fc1-bd2a-11d3951c4b5a None None] MFA Rules not processed for user `733f529b645a40ff99245aed2991b626`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:43.793199 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-b62e4d41-e632-4fc1-bd2a-11d3951c4b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:43.856829 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1447/2902] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:43.861285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02907aa9-3a38-4dbe-bce4-8acb64b8d421 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.907345 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:43.925562 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02907aa9-3a38-4dbe-bce4-8acb64b8d421 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:43.930821 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd923c1-7a8a-4695-ab5a-942e29362d8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:43.931424 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd923c1-7a8a-4695-ab5a-942e29362d8b admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:43.931724 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd923c1-7a8a-4695-ab5a-942e29362d8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:43.932005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd923c1-7a8a-4695-ab5a-942e29362d8b admin admin] PATH_INFO: `/v3/projects` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:43.932641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd923c1-7a8a-4695-ab5a-942e29362d8b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:43.934543 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd923c1-7a8a-4695-ab5a-942e29362d8b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:43.960045 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1456/2903] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:43 2022] POST /identity/v3/projects => generated 373 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:43.964081 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62e4d41-e632-4fc1-bd2a-11d3951c4b5a None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:43.974669 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62e4d41-e632-4fc1-bd2a-11d3951c4b5a None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:44.031952 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79301e6-e2eb-4408-8e3c-0f1cdd6c659b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:44.032483 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79301e6-e2eb-4408-8e3c-0f1cdd6c659b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:44.032767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79301e6-e2eb-4408-8e3c-0f1cdd6c659b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:44.033070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79301e6-e2eb-4408-8e3c-0f1cdd6c659b admin admin] PATH_INFO: `/v3/users` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:44.033708 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79301e6-e2eb-4408-8e3c-0f1cdd6c659b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:44.035726 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79301e6-e2eb-4408-8e3c-0f1cdd6c659b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:44.056630 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1448/2904] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:43 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:44.060616 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd923c1-7a8a-4695-ab5a-942e29362d8b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:44.072051 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd923c1-7a8a-4695-ab5a-942e29362d8b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:44.076713 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7b5458-2c9f-4917-a2ef-cf05d73685fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:44.077310 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7b5458-2c9f-4917-a2ef-cf05d73685fc admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:44.077592 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7b5458-2c9f-4917-a2ef-cf05d73685fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:44.077866 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7b5458-2c9f-4917-a2ef-cf05d73685fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:44.078511 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7b5458-2c9f-4917-a2ef-cf05d73685fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:44.080159 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7b5458-2c9f-4917-a2ef-cf05d73685fc admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:44.091710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1457/2905] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:44.095851 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79301e6-e2eb-4408-8e3c-0f1cdd6c659b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:44.107447 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79301e6-e2eb-4408-8e3c-0f1cdd6c659b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:44.112097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f68c24-dd6e-42bc-be9e-63e6876d0cb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:44.112772 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f68c24-dd6e-42bc-be9e-63e6876d0cb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:44.113053 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f68c24-dd6e-42bc-be9e-63e6876d0cb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:44.113327 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f68c24-dd6e-42bc-be9e-63e6876d0cb8 admin admin] PATH_INFO: `/v3/projects/2a169d4f92e846dba41133edd93c4c5a/users/d545e700d2a745f7ba43239ae54c8d42/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:44.128070 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f68c24-dd6e-42bc-be9e-63e6876d0cb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a169d4f92e846dba41133edd93c4c5a, user_id=d545e700d2a745f7ba43239ae54c8d42, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:44.130721 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f68c24-dd6e-42bc-be9e-63e6876d0cb8 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:44.143789 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1449/2906] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:44 2022] PUT /identity/v3/projects/2a169d4f92e846dba41133edd93c4c5a/users/d545e700d2a745f7ba43239ae54c8d42/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:44.148549 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7b5458-2c9f-4917-a2ef-cf05d73685fc admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:44.158258 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7b5458-2c9f-4917-a2ef-cf05d73685fc admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:44.218013 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd54a0b-a7bb-4b1c-b088-2cc29cd1cff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:44.218612 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd54a0b-a7bb-4b1c-b088-2cc29cd1cff1 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:44.218888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd54a0b-a7bb-4b1c-b088-2cc29cd1cff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:44.219158 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd54a0b-a7bb-4b1c-b088-2cc29cd1cff1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:44.219823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd54a0b-a7bb-4b1c-b088-2cc29cd1cff1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:44.221430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd54a0b-a7bb-4b1c-b088-2cc29cd1cff1 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:44.231206 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1458/2907] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:44 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:44.235387 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f68c24-dd6e-42bc-be9e-63e6876d0cb8 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:44.245616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f68c24-dd6e-42bc-be9e-63e6876d0cb8 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:44.250343 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:44.250998 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:44.251279 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:44.251560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] PATH_INFO: `/v3/projects/2a169d4f92e846dba41133edd93c4c5a/users/d545e700d2a745f7ba43239ae54c8d42/roles/ee6d387ccd8c4db1847085e4aa8b762a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:44.253520 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a169d4f92e846dba41133edd93c4c5a, user_id=d545e700d2a745f7ba43239ae54c8d42, role_id=ee6d387ccd8c4db1847085e4aa8b762a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:44.256208 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:44.267354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1450/2908] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:44 2022] PUT /identity/v3/projects/2a169d4f92e846dba41133edd93c4c5a/users/d545e700d2a745f7ba43239ae54c8d42/roles/ee6d387ccd8c4db1847085e4aa8b762a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:44.279354 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043af35f-c7c4-480f-8cc8-c8dee7498dd9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:44.279641 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043af35f-c7c4-480f-8cc8-c8dee7498dd9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:44.279935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043af35f-c7c4-480f-8cc8-c8dee7498dd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:44.308554 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-043af35f-c7c4-480f-8cc8-c8dee7498dd9 None None] MFA Rules not processed for user `d545e700d2a745f7ba43239ae54c8d42`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:44.309679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-043af35f-c7c4-480f-8cc8-c8dee7498dd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:44.380214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1459/2909] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:44 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:44.392028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:44.393747 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:44.403613 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:44.456616 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f227284-9258-41dc-88d8-db04b2480c46 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:44.460485 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be04d3-2445-45d8-98e0-54c4bcab9df8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:44.461085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be04d3-2445-45d8-98e0-54c4bcab9df8 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:44.461365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be04d3-2445-45d8-98e0-54c4bcab9df8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:44.461638 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be04d3-2445-45d8-98e0-54c4bcab9df8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:44.518580 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be04d3-2445-45d8-98e0-54c4bcab9df8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:44.520039 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be04d3-2445-45d8-98e0-54c4bcab9df8 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:44.529318 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1451/2910] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:44 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:46.027663 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043af35f-c7c4-480f-8cc8-c8dee7498dd9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:46.029108 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-043af35f-c7c4-480f-8cc8-c8dee7498dd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:46.041595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043af35f-c7c4-480f-8cc8-c8dee7498dd9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:46.045086 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa394ef8-49fd-4635-b32a-b7ac8436c823 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:46.045677 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa394ef8-49fd-4635-b32a-b7ac8436c823 service cinder] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:46.045978 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa394ef8-49fd-4635-b32a-b7ac8436c823 service cinder] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:46.046248 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa394ef8-49fd-4635-b32a-b7ac8436c823 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:46.078905 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:46.105774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa394ef8-49fd-4635-b32a-b7ac8436c823 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:46.107213 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa394ef8-49fd-4635-b32a-b7ac8436c823 service cinder] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:46.116934 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1460/2911] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:46 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:46.646920 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be04d3-2445-45d8-98e0-54c4bcab9df8 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:46.656100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be04d3-2445-45d8-98e0-54c4bcab9df8 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:46.659648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3ed4e3-2bef-41fa-aa30-efa40d82bcd6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:46.660263 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ed4e3-2bef-41fa-aa30-efa40d82bcd6 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:46.660540 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ed4e3-2bef-41fa-aa30-efa40d82bcd6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:46.660809 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ed4e3-2bef-41fa-aa30-efa40d82bcd6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:46.669779 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3ed4e3-2bef-41fa-aa30-efa40d82bcd6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:46.671175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3ed4e3-2bef-41fa-aa30-efa40d82bcd6 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:46.681673 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1452/2912] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:46 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:50.552110 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa394ef8-49fd-4635-b32a-b7ac8436c823 service cinder] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:50.614749 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa394ef8-49fd-4635-b32a-b7ac8436c823 service cinder] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:50.619954 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:50.620571 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:50.620848 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:50.621117 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] PATH_INFO: `/v3/users/fff060c5b46a4d479cede3f35cc9a709` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:50.622224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fff060c5b46a4d479cede3f35cc9a709)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:50.624176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:50.670500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fff060c5b46a4d479cede3f35cc9a709'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:50.679216 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fff060c5b46a4d479cede3f35cc9a709'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:50.679865 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fff060c5b46a4d479cede3f35cc9a709'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:50.680477 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fff060c5b46a4d479cede3f35cc9a709'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:50.687410 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fff060c5b46a4d479cede3f35cc9a709'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:50.705363 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1461/2913] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:50 2022] DELETE /identity/v3/users/fff060c5b46a4d479cede3f35cc9a709 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:50.710750 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f1b8f3-7e90-4889-8431-22217bd90be0 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:50.711239 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f1b8f3-7e90-4889-8431-22217bd90be0 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:50.711362 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f1b8f3-7e90-4889-8431-22217bd90be0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:50.729963 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-f0f1b8f3-7e90-4889-8431-22217bd90be0 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:50.731157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f0f1b8f3-7e90-4889-8431-22217bd90be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:50.788213 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1453/2914] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:50.798234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:50.799716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:50.864317 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95292bda-7099-4469-8f30-42c011ef2f85 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:50.868205 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b542b243-d9c1-4c07-b2ef-b8c8df60ea1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:50.868741 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b542b243-d9c1-4c07-b2ef-b8c8df60ea1e service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:50.869020 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b542b243-d9c1-4c07-b2ef-b8c8df60ea1e service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:50.869294 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b542b243-d9c1-4c07-b2ef-b8c8df60ea1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:50.878284 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b542b243-d9c1-4c07-b2ef-b8c8df60ea1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:50.879728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b542b243-d9c1-4c07-b2ef-b8c8df60ea1e service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:50.884099 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:50.891385 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1462/2915] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:28:51.174340 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f1b8f3-7e90-4889-8431-22217bd90be0 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:51.175921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f0f1b8f3-7e90-4889-8431-22217bd90be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:51.189267 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f1b8f3-7e90-4889-8431-22217bd90be0 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:51.195017 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:51.195835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:51.196185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:51.196785 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] PATH_INFO: `/v3/projects/665f2b8696ce494188535f65b8900b1a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:51.198121 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=665f2b8696ce494188535f65b8900b1a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:51.200232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:51.246054 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:51.250028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '665f2b8696ce494188535f65b8900b1a', 'request_id': 'req-46c8c6f2-2842-497e-9b38-cd358f0d18c9'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.257193 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '665f2b8696ce494188535f65b8900b1a', 'request_id': 'req-46c8c6f2-2842-497e-9b38-cd358f0d18c9'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.262879 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '665f2b8696ce494188535f65b8900b1a', 'request_id': 'req-46c8c6f2-2842-497e-9b38-cd358f0d18c9'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.274934 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] The token cache is being invalidate because project 665f2b8696ce494188535f65b8900b1a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:51.275227 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.275900 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.277367 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1454/2916] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:51 2022] DELETE /identity/v3/projects/665f2b8696ce494188535f65b8900b1a => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:28:51.281314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b542b243-d9c1-4c07-b2ef-b8c8df60ea1e service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:51.291492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b542b243-d9c1-4c07-b2ef-b8c8df60ea1e service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:51.353591 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:51.354221 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:51.354500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:51.354803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] PATH_INFO: `/v3/users/733f529b645a40ff99245aed2991b626` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:51.355957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=733f529b645a40ff99245aed2991b626)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:51.357911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:51.403886 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '733f529b645a40ff99245aed2991b626'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.411176 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '733f529b645a40ff99245aed2991b626'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.411817 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '733f529b645a40ff99245aed2991b626'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.412414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '733f529b645a40ff99245aed2991b626'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.420095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '733f529b645a40ff99245aed2991b626'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.437341 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1463/2917] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:51 2022] DELETE /identity/v3/users/733f529b645a40ff99245aed2991b626 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:51.703948 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:51.705425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:51.718425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c8c6f2-2842-497e-9b38-cd358f0d18c9 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:51.775649 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:51.776301 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:51.776583 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:51.776860 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] PATH_INFO: `/v3/projects/8474325d898447e2aa45b1c8400ab6f7` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:51.777971 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8474325d898447e2aa45b1c8400ab6f7)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:51.779906 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:51.826569 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8474325d898447e2aa45b1c8400ab6f7', 'request_id': 'req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.833927 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8474325d898447e2aa45b1c8400ab6f7', 'request_id': 'req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.839764 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8474325d898447e2aa45b1c8400ab6f7', 'request_id': 'req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.850835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] The token cache is being invalidate because project 8474325d898447e2aa45b1c8400ab6f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:51.851125 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.851783 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.853245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1455/2918] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:51 2022] DELETE /identity/v3/projects/8474325d898447e2aa45b1c8400ab6f7 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:51.857377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:51.858774 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:51.871899 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6b4faf-12ce-42d4-b73e-60996cae06d8 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:51.930312 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:51.930933 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:51.931214 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:51.931489 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] PATH_INFO: `/v3/users/d545e700d2a745f7ba43239ae54c8d42` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:51.932657 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=d545e700d2a745f7ba43239ae54c8d42)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:51.934585 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:51.955581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:51.980716 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd545e700d2a745f7ba43239ae54c8d42'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.987582 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd545e700d2a745f7ba43239ae54c8d42'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.988224 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd545e700d2a745f7ba43239ae54c8d42'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.988836 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd545e700d2a745f7ba43239ae54c8d42'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:51.996594 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7895eb78-8dc9-4c0b-b3cc-514497c763aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd545e700d2a745f7ba43239ae54c8d42'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:52.014268 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1464/2919] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:28:51 2022] DELETE /identity/v3/users/d545e700d2a745f7ba43239ae54c8d42 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:52.524153 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:52.525676 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:52.539515 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10763d69-4ee7-4a4b-b3ea-23384fb7cc19 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:52.597593 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:52.598145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:52.598430 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:52.598703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] PATH_INFO: `/v3/projects/2a169d4f92e846dba41133edd93c4c5a` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:52.599863 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a169d4f92e846dba41133edd93c4c5a)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:52.601813 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:52.604040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:52.650819 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a169d4f92e846dba41133edd93c4c5a', 'request_id': 'req-e07d4818-fd3c-4c7b-b286-76040cff24de'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:52.657837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a169d4f92e846dba41133edd93c4c5a', 'request_id': 'req-e07d4818-fd3c-4c7b-b286-76040cff24de'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:52.663476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a169d4f92e846dba41133edd93c4c5a', 'request_id': 'req-e07d4818-fd3c-4c7b-b286-76040cff24de'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:52.674427 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] The token cache is being invalidate because project 2a169d4f92e846dba41133edd93c4c5a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:28:52.674720 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:52.675350 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e07d4818-fd3c-4c7b-b286-76040cff24de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:28:52.676846 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1456/2920] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:28:52 2022] DELETE /identity/v3/projects/2a169d4f92e846dba41133edd93c4c5a => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 15 14:28:52.682361 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32ddd0-47dd-4266-b995-ffce941b27b9 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:52.682651 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32ddd0-47dd-4266-b995-ffce941b27b9 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:52.682925 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32ddd0-47dd-4266-b995-ffce941b27b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:52.702858 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-6d32ddd0-47dd-4266-b995-ffce941b27b9 None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:52.704104 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6d32ddd0-47dd-4266-b995-ffce941b27b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:52.762752 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1465/2921] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:52.770420 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a69f73-6561-48be-bdbd-8aa371164c5c None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:52.770913 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a69f73-6561-48be-bdbd-8aa371164c5c None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:52.771015 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a69f73-6561-48be-bdbd-8aa371164c5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:52.791619 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-84a69f73-6561-48be-bdbd-8aa371164c5c None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:52.792912 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-84a69f73-6561-48be-bdbd-8aa371164c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:52.801016 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1457/2922] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:52.805215 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d32ddd0-47dd-4266-b995-ffce941b27b9 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:52.814927 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d32ddd0-47dd-4266-b995-ffce941b27b9 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:52.819861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:52.820464 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:52.820751 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:52.821022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] PATH_INFO: `/v3/domains` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:52.821957 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81655) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 15 14:28:52.822262 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:52.823935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:52.835803 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1466/2923] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 15 14:28:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 15 14:28:52.840084 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a69f73-6561-48be-bdbd-8aa371164c5c None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:52.849334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a69f73-6561-48be-bdbd-8aa371164c5c None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:52.854457 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d75f77-943d-4db3-adbd-09bfef6cdd2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:52.855045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d75f77-943d-4db3-adbd-09bfef6cdd2b admin admin] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:52.855323 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d75f77-943d-4db3-adbd-09bfef6cdd2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:52.855595 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d75f77-943d-4db3-adbd-09bfef6cdd2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:52.856252 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d75f77-943d-4db3-adbd-09bfef6cdd2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:52.858133 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d75f77-943d-4db3-adbd-09bfef6cdd2b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:52.885045 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1458/2924] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 15 14:28:52 2022] POST /identity/v3/projects => generated 369 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:52.889656 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:52.901254 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b47781d-31be-4e38-bd3c-94c0f5da8595 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:52.956286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bc0655-1c8b-4832-9f0b-3a67d044a3f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:52.956880 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bc0655-1c8b-4832-9f0b-3a67d044a3f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:52.957160 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bc0655-1c8b-4832-9f0b-3a67d044a3f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:52.957436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bc0655-1c8b-4832-9f0b-3a67d044a3f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:52.958068 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bc0655-1c8b-4832-9f0b-3a67d044a3f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:52.960611 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bc0655-1c8b-4832-9f0b-3a67d044a3f5 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:52.982164 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1467/2925] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 15 14:28:52 2022] POST /identity/v3/users => generated 285 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 15 14:28:52.986560 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d75f77-943d-4db3-adbd-09bfef6cdd2b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:52.996991 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d75f77-943d-4db3-adbd-09bfef6cdd2b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:53.001622 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236c2abd-51d5-4bec-afd3-5c16abbc4790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:53.002221 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c2abd-51d5-4bec-afd3-5c16abbc4790 admin admin] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:53.002501 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c2abd-51d5-4bec-afd3-5c16abbc4790 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:53.002775 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c2abd-51d5-4bec-afd3-5c16abbc4790 admin admin] PATH_INFO: `/v3/roles` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:53.003421 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236c2abd-51d5-4bec-afd3-5c16abbc4790 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:53.005042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236c2abd-51d5-4bec-afd3-5c16abbc4790 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:53.015609 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1459/2926] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:53.019505 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bc0655-1c8b-4832-9f0b-3a67d044a3f5 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:53.030314 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bc0655-1c8b-4832-9f0b-3a67d044a3f5 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:53.034790 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b476122f-e45a-4ba3-99f6-c19277239e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:53.035369 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476122f-e45a-4ba3-99f6-c19277239e17 admin admin] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:53.035647 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476122f-e45a-4ba3-99f6-c19277239e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:53.035949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476122f-e45a-4ba3-99f6-c19277239e17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:53.036581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b476122f-e45a-4ba3-99f6-c19277239e17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:53.038457 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b476122f-e45a-4ba3-99f6-c19277239e17 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:53.048757 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1468/2927] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 15 14:28:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 15 14:28:53.052736 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236c2abd-51d5-4bec-afd3-5c16abbc4790 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:53.062636 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236c2abd-51d5-4bec-afd3-5c16abbc4790 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:53.067204 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:53.067888 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] REQUEST_METHOD: `PUT` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:53.068172 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:53.068451 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] PATH_INFO: `/v3/projects/846cc930ba73441e934b7e9106dbb100/users/17a27c79667c4eb7aad0c4809f719820/roles/b0e7999c9d0d472195e1c5dc6f4d4978` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:53.087123 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] RBAC: Authorizing `identity:create_grant(project_id=846cc930ba73441e934b7e9106dbb100, user_id=17a27c79667c4eb7aad0c4809f719820, role_id=b0e7999c9d0d472195e1c5dc6f4d4978)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:53.090112 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:53.102915 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1460/2928] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 15 14:28:53 2022] PUT /identity/v3/projects/846cc930ba73441e934b7e9106dbb100/users/17a27c79667c4eb7aad0c4809f719820/roles/b0e7999c9d0d472195e1c5dc6f4d4978 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:28:53.114285 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c458ac2-1d7e-457b-98b0-dc18e86ba2da None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:53.114569 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c458ac2-1d7e-457b-98b0-dc18e86ba2da None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:53.114840 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c458ac2-1d7e-457b-98b0-dc18e86ba2da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:53.122405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:28:53.144888 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-7c458ac2-1d7e-457b-98b0-dc18e86ba2da None None] MFA Rules not processed for user `17a27c79667c4eb7aad0c4809f719820`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:28:53.146078 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7c458ac2-1d7e-457b-98b0-dc18e86ba2da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:53.209095 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1469/2929] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:28:53 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:28:53.221933 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:28:53.223585 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:28:53.304425 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b156bb9-95db-4f9a-aa54-3235773bf16b admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:28:53.308100 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d5488d-6927-4172-b8a8-42bcf4396f9a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '54db2e6805fd40f79c03eb3b2319f2b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:28:53.308703 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5488d-6927-4172-b8a8-42bcf4396f9a service glance] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:28:53.308983 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5488d-6927-4172-b8a8-42bcf4396f9a service glance] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:28:53.309255 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5488d-6927-4172-b8a8-42bcf4396f9a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:28:53.317875 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d5488d-6927-4172-b8a8-42bcf4396f9a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:28:53.320146 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d5488d-6927-4172-b8a8-42bcf4396f9a service glance] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:28:53.329182 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1461/2930] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:28:53 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:29:03.206113 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=31 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:29:03.325230 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=85,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:29:36.250530 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c458ac2-1d7e-457b-98b0-dc18e86ba2da None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:29:36.312413 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c458ac2-1d7e-457b-98b0-dc18e86ba2da None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:29:36.316971 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:29:36.317595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:29:36.317872 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:29:36.318142 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] PATH_INFO: `/v3/users/17a27c79667c4eb7aad0c4809f719820` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:29:36.319234 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] RBAC: Authorizing `identity:delete_user(user_id=17a27c79667c4eb7aad0c4809f719820)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:29:36.321182 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:29:36.368109 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17a27c79667c4eb7aad0c4809f719820'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.375213 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17a27c79667c4eb7aad0c4809f719820'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.375861 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17a27c79667c4eb7aad0c4809f719820'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.376470 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17a27c79667c4eb7aad0c4809f719820'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.383414 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17a27c79667c4eb7aad0c4809f719820'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.400967 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1470/2931] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:29:36 2022] DELETE /identity/v3/users/17a27c79667c4eb7aad0c4809f719820 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:29:36.405537 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b5f5c-cd12-45ea-95cf-0fc8e0c115ef None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:29:36.406048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b5f5c-cd12-45ea-95cf-0fc8e0c115ef None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:29:36.406157 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b5f5c-cd12-45ea-95cf-0fc8e0c115ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:29:36.425572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-e97b5f5c-cd12-45ea-95cf-0fc8e0c115ef None None] MFA Rules not processed for user `22bd926197084ae38005e05ce6f936ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:29:36.426802 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e97b5f5c-cd12-45ea-95cf-0fc8e0c115ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:29:36.485921 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1462/2932] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:29:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 15 14:29:36.496288 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:29:36.497725 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:29:36.553907 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2c5bd7-0fad-4357-808e-86f8fd56e58b admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:29:36.558213 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:29:36.558805 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:29:36.559084 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:29:36.559357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:29:36.568245 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:29:36.569667 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:29:36.579536 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1471/2933] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:29:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:29:36.845396 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97b5f5c-cd12-45ea-95cf-0fc8e0c115ef None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:29:36.846812 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e97b5f5c-cd12-45ea-95cf-0fc8e0c115ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:29:36.860590 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97b5f5c-cd12-45ea-95cf-0fc8e0c115ef None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:29:36.865181 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:29:36.865814 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:29:36.866591 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:29:36.867040 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] PATH_INFO: `/v3/projects/846cc930ba73441e934b7e9106dbb100` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:29:36.868334 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=846cc930ba73441e934b7e9106dbb100)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:29:36.870776 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:29:36.918114 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '846cc930ba73441e934b7e9106dbb100', 'request_id': 'req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.926085 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '846cc930ba73441e934b7e9106dbb100', 'request_id': 'req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.931567 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '846cc930ba73441e934b7e9106dbb100', 'request_id': 'req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.943191 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] The token cache is being invalidate because project 846cc930ba73441e934b7e9106dbb100 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81654) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:29:36.943479 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.944142 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:29:36.945586 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1463/2934] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:29:36 2022] DELETE /identity/v3/projects/846cc930ba73441e934b7e9106dbb100 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:29:46.575482 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:29:46.940203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:29:56.049823 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:29:56.051195 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:29:56.120728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ccf9e-cdcb-455f-abec-59fcdf4a5f35 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:29:56.180780 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:29:56.181386 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:29:56.181669 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:29:56.181942 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:29:56.183648 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:29:56.195325 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:29:56.196940 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:29:56.211465 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1472/2935] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:29:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:30:00.120884 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:30:00.122374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:30:00.179175 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6e7ef3-5273-48b6-aca5-944026d1b8f2 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:30:00.182728 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc809ab-816a-427d-9eeb-ae0873658b87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:30:00.183266 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc809ab-816a-427d-9eeb-ae0873658b87 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:30:00.183543 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc809ab-816a-427d-9eeb-ae0873658b87 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:30:00.183847 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc809ab-816a-427d-9eeb-ae0873658b87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:30:00.185512 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-2bc809ab-816a-427d-9eeb-ae0873658b87 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:30:00.272785 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc809ab-816a-427d-9eeb-ae0873658b87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:30:00.274202 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc809ab-816a-427d-9eeb-ae0873658b87 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:30:00.293805 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1464/2936] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:30:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:30:01.483577 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:30:01.485672 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:30:01.497090 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be3c694-6fff-4ec8-950b-924618c9d879 service placement] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:30:01.500890 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:30:01.501486 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:30:01.501768 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:30:01.502042 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:30:01.503728 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:30:01.517167 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:30:01.518595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:30:01.529617 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1473/2937] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:30:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:30:03.836111 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd81763-691d-4243-a6f0-816c15b5de83 None None] REQUEST_METHOD: `POST` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:30:03.836573 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd81763-691d-4243-a6f0-816c15b5de83 None None] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:30:03.836695 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd81763-691d-4243-a6f0-816c15b5de83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:30:03.855455 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.auth.core [None req-1cd81763-691d-4243-a6f0-816c15b5de83 None None] MFA Rules not processed for user `1c072306d6534925b92fbc5ef01aaaf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81654) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:30:03.856658 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1cd81763-691d-4243-a6f0-816c15b5de83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:30:03.909626 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1465/2938] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:30:03 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:30:11.524710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=64 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:30:13.903004 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=91 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:32:05.750944 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:32:05.759959 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c3fabe-4333-4739-8592-1e821070dd73 service neutron] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:32:05.763583 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea13cd43-e583-404c-a189-4d33dd463e62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:32:05.764209 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea13cd43-e583-404c-a189-4d33dd463e62 service neutron] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:32:05.764491 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea13cd43-e583-404c-a189-4d33dd463e62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:32:05.764765 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea13cd43-e583-404c-a189-4d33dd463e62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:32:05.766461 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-ea13cd43-e583-404c-a189-4d33dd463e62 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:05.775935 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea13cd43-e583-404c-a189-4d33dd463e62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:32:05.777357 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea13cd43-e583-404c-a189-4d33dd463e62 service neutron] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:32:05.785372 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1474/2939] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:32:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:32:12.065960 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd81763-691d-4243-a6f0-816c15b5de83 None None] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:32:12.067354 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-1cd81763-691d-4243-a6f0-816c15b5de83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:12.124682 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd81763-691d-4243-a6f0-816c15b5de83 None None] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:32:12.128205 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f54979e322484a42bb57d1b3a619652c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:32:12.128808 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:32:12.129089 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:32:12.129365 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:32:12.131069 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:12.141780 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:32:12.143185 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:32:12.152565 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1466/2940] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:32:12 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:32:15.781683 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:32:22.147887 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=26 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:32:23.186418 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d51b621-04ff-4e30-8bd3-22c2c2703240 None None] REQUEST_METHOD: `POST` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:32:23.186924 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d51b621-04ff-4e30-8bd3-22c2c2703240 None None] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:32:23.187043 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d51b621-04ff-4e30-8bd3-22c2c2703240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:32:23.206242 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.auth.core [None req-9d51b621-04ff-4e30-8bd3-22c2c2703240 None None] MFA Rules not processed for user `1c072306d6534925b92fbc5ef01aaaf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81655) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 15 14:32:23.207442 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9d51b621-04ff-4e30-8bd3-22c2c2703240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:23.215056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1475/2941] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 15 14:32:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 15 14:32:23.986696 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:32:23.988176 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:24.003988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cbedcc-caca-40f6-ac02-698cb177e916 service cinder] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:32:24.008911 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:32:24.009511 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:32:24.009793 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:32:24.010068 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:32:24.011978 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:24.023724 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:32:24.025290 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:32:24.036109 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1467/2942] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:32:23 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:32:33.200693 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=3 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:32:34.031656 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:32:37.574125 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d51b621-04ff-4e30-8bd3-22c2c2703240 None None] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:32:37.575992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9d51b621-04ff-4e30-8bd3-22c2c2703240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:37.592537 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d51b621-04ff-4e30-8bd3-22c2c2703240 None None] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:32:37.598005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:32:37.598608 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:32:37.598884 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:32:37.599150 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:32:37.600911 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:37.613260 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:32:37.614855 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:32:37.626257 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1476/2943] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:32:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:32:42.539345 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:32:42.540794 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:42.556380 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffe2559-4a88-458d-bfac-5b6c1e9a6ca5 service placement] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:32:42.562582 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:32:42.563178 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:32:42.563456 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:32:42.563754 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:32:42.565510 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:32:42.578246 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:32:42.579834 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:32:42.591158 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1468/2944] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:32:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:32:47.621320 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:32:52.586659 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=6 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:34:34.784358 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:34:34.797174 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7780ce79-7f1d-43e7-b627-ceefb4b22931 service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:34:34.803764 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce6744790db74f0aaffc6eb269bf6737', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:34:34.804380 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:34:34.804666 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:34:34.804965 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:34:34.806801 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:34:34.821538 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:34:34.823981 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:34:34.835462 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1477/2945] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:34:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:34:44.422496 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:34:44.432107 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9914ca5e-28d2-4803-8d24-19481aa5cf89 service nova] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:34:44.436356 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '922012eacf244f9480c180c2eb9eaf28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:34:44.436950 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:34:44.437232 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:34:44.437506 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:34:44.439276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:34:44.503097 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:34:44.504602 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:34:44.514138 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1469/2946] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:34:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:34:44.830655 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=6 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:34:47.568492 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:34:47.582581 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44f2fb5-2eca-4210-96c1-e4a45c8f58ab service nova] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:34:47.589920 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:34:47.590161 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:34:47.590436 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:34:47.590710 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:34:47.592819 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:34:47.605947 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:34:47.607595 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:34:47.620845 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1478/2947] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:34:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:34:50.791130 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:34:50.792988 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:34:50.806042 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd88f051-828d-4f24-aa5b-9b523dfb44c9 service neutron] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:34:50.810855 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:34:50.811478 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:34:50.811792 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:34:50.812067 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] PATH_INFO: `/v3/users/1c072306d6534925b92fbc5ef01aaaf6` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:34:50.813245 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c072306d6534925b92fbc5ef01aaaf6)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:34:50.815148 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:34:50.868044 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c072306d6534925b92fbc5ef01aaaf6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:50.876254 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c072306d6534925b92fbc5ef01aaaf6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:50.876870 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c072306d6534925b92fbc5ef01aaaf6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:50.877476 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c072306d6534925b92fbc5ef01aaaf6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:50.884374 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c072306d6534925b92fbc5ef01aaaf6'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:50.902470 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1470/2948] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:34:50 2022] DELETE /identity/v3/users/1c072306d6534925b92fbc5ef01aaaf6 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:34:51.075956 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:34:51.088542 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd78561-432f-460f-8ea1-4f638b511c9a service placement] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:34:51.145375 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:34:51.146005 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:34:51.146286 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:34:51.146561 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] PATH_INFO: `/v3/projects/b9f4deb43acf410aa0a8e88c20ddb3f5` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:34:51.147717 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9f4deb43acf410aa0a8e88c20ddb3f5)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:34:51.149632 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:34:51.195939 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9f4deb43acf410aa0a8e88c20ddb3f5', 'request_id': 'req-7ac5232d-6a96-4e0c-aa26-677427cbcf04'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.203949 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9f4deb43acf410aa0a8e88c20ddb3f5', 'request_id': 'req-7ac5232d-6a96-4e0c-aa26-677427cbcf04'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.209431 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9f4deb43acf410aa0a8e88c20ddb3f5', 'request_id': 'req-7ac5232d-6a96-4e0c-aa26-677427cbcf04'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.221103 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] The token cache is being invalidate because project b9f4deb43acf410aa0a8e88c20ddb3f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:34:51.221396 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.222039 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.223519 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1479/2949] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:34:51 2022] DELETE /identity/v3/projects/b9f4deb43acf410aa0a8e88c20ddb3f5 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:34:51.227464 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:34:51.228898 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:34:51.242242 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f789c-9eff-4217-8409-d12690441ffb admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:34:51.300243 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:34:51.300880 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:34:51.301159 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:34:51.301429 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] PATH_INFO: `/v3/users/0268010872084a15bc890ba2bc431e02` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:34:51.302572 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0268010872084a15bc890ba2bc431e02)` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:34:51.304529 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:34:51.310835 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:34:51.359140 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0268010872084a15bc890ba2bc431e02'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.367195 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0268010872084a15bc890ba2bc431e02'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.368028 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0268010872084a15bc890ba2bc431e02'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.369012 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0268010872084a15bc890ba2bc431e02'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.377389 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.notifications [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0268010872084a15bc890ba2bc431e02'} {{(pid=81654) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.397006 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1471/2950] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 15 14:34:51 2022] DELETE /identity/v3/users/0268010872084a15bc890ba2bc431e02 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:34:51.571500 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:34:51.573813 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:34:51.586620 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac5232d-6a96-4e0c-aa26-677427cbcf04 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:34:51.643430 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22bd926197084ae38005e05ce6f936ab', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f973480c94d94371b4cef62cd36d8410', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:34:51.644092 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:34:51.644368 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:34:51.644637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] PATH_INFO: `/v3/projects/16e1410c94744608a40b482c96aca5f7` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:34:51.645739 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] RBAC: Authorizing `identity:delete_project(project_id=16e1410c94744608a40b482c96aca5f7)` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:34:51.647637 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:34:51.669056 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:34:51.694692 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16e1410c94744608a40b482c96aca5f7', 'request_id': 'req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.702992 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16e1410c94744608a40b482c96aca5f7', 'request_id': 'req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.708586 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16e1410c94744608a40b482c96aca5f7', 'request_id': 'req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311'} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.720731 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] The token cache is being invalidate because project 16e1410c94744608a40b482c96aca5f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81655) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 15 14:34:51.721022 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.721644 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.notifications [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81655) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 15 14:34:51.723105 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1480/2951] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 15 14:34:51 2022] DELETE /identity/v3/projects/16e1410c94744608a40b482c96aca5f7 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 15 14:34:58.034837 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] Authenticating user token {{(pid=81654) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:34:58.036325 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:34:58.105276 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9628404f-8dc5-400c-b8a4-88de2e222746 admin admin] Validating token access rules against request {{(pid=81654) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:34:58.163145 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f734e4da-1f8f-4110-9137-886e70954349 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81654) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:34:58.163767 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734e4da-1f8f-4110-9137-886e70954349 service placement] REQUEST_METHOD: `GET` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:34:58.164048 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734e4da-1f8f-4110-9137-886e70954349 service placement] SCRIPT_NAME: `/identity` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:34:58.164317 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734e4da-1f8f-4110-9137-886e70954349 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81654) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:34:58.165992 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.fernet_utils [None req-f734e4da-1f8f-4110-9137-886e70954349 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81654) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:34:58.177648 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f734e4da-1f8f-4110-9137-886e70954349 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:34:58.179203 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f734e4da-1f8f-4110-9137-886e70954349 service placement] RBAC: Authorization granted {{(pid=81654) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:34:58.188871 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: [pid: 81654|app: 0|req: 1472/2952] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:34:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:35:01.717322 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:35:08.184640 dv-cinder-856971-2-iscsi devstack@keystone.service[81654]: DEBUG dbcounter [-] [81654] Writing DB stats keystone:SELECT=81,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81654) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 15 14:40:02.043838 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] Authenticating user token {{(pid=81655) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 15 14:40:02.045226 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.fernet_utils [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81655) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 15 14:40:02.133887 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2f4d4-03f4-4d5a-bdae-3bd9235ea311 admin admin] Validating token access rules against request {{(pid=81655) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 15 14:40:02.139459 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d7663a-ef44-40a2-b9b9-2742db52079d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '62299b21a55c46d78a62098ee9e3c497', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '64eda23ae35b454786569cfb259d972b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81655) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 15 14:40:02.140122 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7663a-ef44-40a2-b9b9-2742db52079d service placement] REQUEST_METHOD: `GET` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 15 14:40:02.140405 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7663a-ef44-40a2-b9b9-2742db52079d service placement] SCRIPT_NAME: `/identity` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 15 14:40:02.140679 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7663a-ef44-40a2-b9b9-2742db52079d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81655) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 15 14:40:02.150207 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d7663a-ef44-40a2-b9b9-2742db52079d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 15 14:40:02.151878 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d7663a-ef44-40a2-b9b9-2742db52079d service placement] RBAC: Authorization granted {{(pid=81655) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 15 14:40:02.163377 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: [pid: 81655|app: 0|req: 1481/2953] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 15 14:40:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 15 14:40:12.158963 dv-cinder-856971-2-iscsi devstack@keystone.service[81655]: DEBUG dbcounter [-] [81655] Writing DB stats keystone:SELECT=37 {{(pid=81655) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}